| Age | Commit message (Collapse) | Author |
|---|
|
With introduction of DRILL-5735 , the descriptions for about half the system options still remain missing. This commit collects descriptions review by @bbevens
1. Update options for HashAgg/Join (@Ben-Zvi )
2. Update options for Parquet Reader/Writer (@sachouche )
3. Update options for Planners (@HanumathRao , @vdiravka , @KazydubB )
4. Update options for BatchSizing (@bitblender )
5. Update options for Planner Optimizations (@arina-ielchiieva )
6. Update options for Security & Kafka (Krystal Nguyen)
7. Update options for Misc entries (@arina-ielchiieva , @vvysotskyi )
In additional, there is a patch for `org.apache.drill.exec.compile.ClassTransformer.scalar_replacement` , which appears to have replaced `exec.compile.scalar_replacement`. References to the latter have been removed to avoid confusion.
Additional changes include moving the `ClassTransformer` validator to `ExecConstants.java`
Adding support for internal options' descriptions
Removed mention of {{Will be removed in 1.15.0}}. (Refer DRILL-6527)
|
|
Allow search/sort in the Options webUI
Additional details, like the description is provided as well in a JScript lookup map. This helps reduce the need for the server to constantly recreate the entire page with the description details, as the client browser can fill in these details.
Developers will be expected to update the description as old/new options are introduced or deprecated.
System table options_val also carries the description, albeit a 'shorter' version.
closes #1279
|
|
|
|
|
|
This change includes:
DRILL-5783:
- A unit test is created for the priority queue in the TopN operator.
- The code generation classes passed around a completely unused function registry reference in some places so it is removed.
- The priority queue had unused parameters for some of its methods so it is removed.
DRILL-5841:
- Created standardized temp directory classes DirTestWatcher, SubDirTestWatcher, and BaseDirTestWatcher. And updated all unit tests to use them.
DRILL-5894:
- Removed the dfs_test storage plugin for tests and replaced it with the already existing dfs storage plugin.
Misc:
- General code cleanup.
- Removed unnecessary use of String.format in the tests.
This closes #984
|
|
batch has large number of fields.
- Changed estimation of max index value and added comments.
close #818
|
|
This PR is purely a refactoring: no functionality is added or changed.
The refactoring splits various context and related classes into a set
of new interfaces with needed for operator-level unit tests. The other,
Drillbit-related methods are left in the original interfaces. Most code
need not change.
The changes here allow operator-level unit tests to mock up the
exec-time methods so we can use them without firing up a Drillbit (or
using mocking libraries).
A later PR will provide the sub-operator test framework that uses this
refactoring.
Changes include:
* The OptionManager is split, with read-only methods moving to a new
OptionSet interface.
* The FragmentContext is split, with an exec-only FragmentExecContext
proving low-level methods.
* OperatorStats is split, with a new OperatorStatReceiver class
providing write-only support to operators.
* Several places that accepted an OperatorContext or FragmentContext,
but needed only an allocator, are changed to accept the allocator
directly.
Includes fixes for code review comments
Adds more comments. Postpones the suggested rename until all affected
code is in master, else it will be difficult to synchronize the rename
across multiple branches.
|
|
DRILL-5052 added the ability to debug generated code. The reviewer suggested
permitting the technique to be used for all Drill operators. This PR provides
the required fixes. Most were small changes, others dealt with the rather
clever way that the existing byte-code merge converted static nested classes
to non-static inner classes, with the way that constructors were inserted
at the byte-code level and so on. See the JIRA for the details.
This code passed the unit tests twice: once with the traditional byte-code
manipulations, a second time using "plain-old Java" code compilation.
Plain-old Java is turned off by default, but can be turned on for all
operators with a single config change: see the JIRA for info. Consider
the plain-old Java option to be experimental: very handy for debugging,
perhaps not quite tested enough for production use.
close apache/drill#716
|
|
Provides a second compilation path for generated code: “plan old Java”
in which generated code inherit from their templates. Such code can be
compiled directly, allowing easy debugging of generated code.
Also show to generate two classes in the External Sort Batch as “plain
old Java” to enable IDE debugging of that generated code. Required
minor clean-up of the templates.
Fixes some broken toString( ) methods in code generation classes
Fixes a variety of small compilation warnings
Adds Java doc to a few classes
Includes clean-up from code review comments.
close apache/drill#660
|
|
Closes #394
|
|
|
|
through ClassReader.EXPAND_FRAMES to satisfy complaint from ASM - rationalize AsmUtils methods' argument lists
TestBugFixes (in ...drill.jdbc.test)
- created this to hold random bug fix tests
|
|
Foreman/WorkManager in order to ensure consistent handling, and avoid hangs and races, with the goal of improving Drillbit robustness.
I did my best to keep these clean when I split them up, but this core commit
may depend on some minor changes in the hygiene commit that is also
associated with this bug, so either both should be applied, or neither.
The core commit should be applied first.
protocol/pom.xml
- updated protocol buffer compiler version to 2.6
- this made slight modifications to the formats of a few committed protobuf
files
AutoCloseables
- created org.apache.drill.common.AutoCloseables to handle closing these
quietly
BaseTestQuery, and derivatives
- factored out pieces into QueryTestUtil so they can be reused
DeferredException:
- created this so we can collect exceptions during the shutdown process
Drillbit
- uses AutoCloseables for the WorkManager and for the storeProvider
- allow start() to take a RemoteServiceSet
- private, final, formatting
Foreman
- added new state CANCELLATION_REQUESTED (via UserBitShared.proto) to represent
the time between request of a cancellation, and acknowledgement from all
remote endpoints running fragments on a query's behalf
- created ForemanResult to manage interleaving cleanup effects/failure with
query result state
- does not need to implement Comparable
- does not need to implement Closeable
- thread blocking fixes
- add resultSent flag
- add code to log plan fragments with endpoint assignments
- added finals, cleaned up formatting
- do queue management in acquireQuerySemaphore; local tests pass
- rename getContext() to getQueryContext()
- retain DrillbitContext
- a couple of exception injections for testing
- minor formatting
- TODOs
FragmentContext
- added a DeferredException to collect errors during startup/shutdown sequences
FragmentExecutor
- eliminated CancelableQuery
- use the FragmentContext's DeferredException for errors
- common subexpression elimination
- cleaned up
QueryContext
- removed unnecessary functions (with some outside classes tweaked for this)
- finals, formatting
QueryManager
- merge in QueryStatus
- affects Foreman, ../batch/ControlHandlerImpl,
and ../../server/rest/ProfileResources
- made some methods private
- removed unused imports
- add finals and formatting
- variable renaming to improve readability
- formatting
- comments
- TODOs
QueryStatus
- getAsInfo() private
- member renaming
- member access changes
- formatting
- TODOs
QueryTestUtil, BaseTestQuery, TestDrillbitResilience
- make maxWidth a parameter to server startup
SelfCleaningRunnable
- created org.apache.drill.common.SelfCleaningRunnable
SingleRowListener
- created org.apache.drill.SingleRowListener results listener
- use in TestDrillbitResilience
TestComparisonFunctions
- fix not to close the FragmentContext multiple times
TestDrillbitResilience
- created org.apache.drill.exec.server.TestDrillbitResilience to test drillbit
resilience in the face of exceptions and failures during queries
TestWithZookeeper
- factor out work into ZookeeperHelper so that it can be reused by
TestDrillbitResilience
UserBitShared
- get rid of unused UNKNOWN_QUERY
WorkEventBus
- rename methods, affects Foreman and ControlHandlerImpl
- remove unused WorkerBee reference
- most members final
- formatting
WorkManager
- Closeable to AutoCloseable
- removed unused incomingFragments Set
- eliminated unnecessary eventThread and pendingTasks by posting Runnables
directly to executor
- use SelfCleaningRunnable for Foreman management
- FragmentExecutor management uses SelfCleaningRunnable
- runningFragments to be a ConcurrentHashMap; TestTpchDistributed passes
- other improvements due to bee no longer needed in various places
- most members final
- minor formatting
- comments
- TODOs
(*) Created exception injection classes to simulate exceptions for testing
- ExceptionInjection
- ExceptionInjector
- ExceptionInjectionUtil
- TestExceptionInjection
DRILL-2245-hygiene: General code cleanup encountered while working on the rest
of this commit. This includes
- making members final whenever possible
- making members private whenever possible
- making loggers private
- removing unused imports
- removing unused private functions
- removing unused public functions
- removing unused local variables
- removing unused private members
- deleting unused files
- cleaning up formatting
- adding spaces before braces in conditionals and loop bodies
- breaking up overly long lines
- removing extra blank lines
While I tried to keep this clean, this commit may have minor dependencies on
DRILL-2245-core that I missed. The intention is just to break this up for
review purposes. Either both commits should be applied, or neither.
|
|
Cleaned up option handling. This includes using finals, making member variables
private whenever possible, and some formatting.
- fixed a bug in the string formatting for the double range validator
- OptionValidator, OptionValue, and their implementations now conspire not to
allow the creation of malformed options because the OptionType has been added
to validator calls to handle OptionValues that are created on demand.
Started with updated byte code rewrite from Jacques
Fixed several problems with scalar value replacement:
- use consistent ASM api version throughout
- stop using deprecated ASM methods (actually causes bugs)
- visitMethodInsn()
- added a couple of missing super.visitEnd()s
- fixed a couple of minor FindBugs issues
- accounted for required stack size increases when replacing holders for
longs and doubles
- added accounting for frame offsets to cope with long and double local
variables and value holder members
- fixed a few minor bugs found with FindBugs
- stop using carrotlabs' hash map lget() method on shared constant data
- fixed an incorrect use of DUP2 on objectrefs when copying long or double
holder members into locals
- fixed a problem with redundant POP instructions left behind after replacement
- fixed a problem with incorrect DUPs in multiple assignment statements
- fixed a problem with DUP_X1 replacement when handling constants in multiple
assignment statements
- fixed a problem with non-replaced holder member post-decrements
- don't replace holders passed to static functions as "out" parameters
(common with Accessors on repeated value vectors)
- increased the maximum required stack size when transferring holder members to
locals
- changed the code generation block type mappings for constants for external
sorts
- fixed problems handling constant and non-constant member variables in
operator classes
- in general, if a holder is assigned to or from an operator member variable,
it can't be replaced (at least not until we replace those as well)
- Use a derived ASM Analyzer (MethodAnalyzer) and Frame
(AssignmentTrackingFrame) in order to establish relationships between
assignments of holders through chains of local variables. This effectively
back-propagates non-replaceability attributes so that if a holder variable
that can't be replaced is assigned to from another holder variable, that
second one cannot be replaced either, and so on through longer chains of
assignments.
- code for dumping generated source code
- MergeAdapter dumps before and after results of scalar replacement
(if it's on)
- fixed some problems in ReplacingBasicValue by replacing HashSet with
IdentityHashMap
- made loggers private
- added a retry strategy for scalar replacement
if a scalar replacement code rewriting fails, then this will try to
regenerate the bytecode again without the scalar replacement.
- bytecode verification is always on now (required for the retry strategy)
- use system option to determine whether scalar replacement should be used
- default option: if scalar replacement fails, retry without it
- force replacement on or off
- unit tests for the retry strategy are based on a single known failure case,
covered by DRILL-2326.
- add tests TestConvertFunctions to test the three scalar replacement options
for the failing test case (testVarCharReturnTripConvertLogical)
- made it possible to set a SYSTEM option as a java property in Drillbit
- added a command line argument to force scalar replacement to be on during
testing in the rootmost pom.xml
In the course of this, added increased checking of intermediate stages of code
rewriting, as well as logging of classes that cause failures.
- work around a bug in ASM's CheckClassAdapter that doesn't allow for checking
of inner classes
Added comments, tidied up formatting, and added "final" in a number of places.
Signed-off-by: vkorukanti <venki.korukanti@gmail.com>
|
|
PlanFragment distribution
Includes:
- Remove dependency on Infinispan
- Update initialize fragments to send in batches.
- Update RPC layer to capture UserRpcExceptions and propagate back.
- Send full stack trace in DrillPBError and let foreman node decide on formatting.
- Increment control rpc version
- Update systables to report current drillbit and version
|
|
without braces
|
|
Add @Inject DrillBuf
Move comparison functions to memory sensitive ones
Add scalar replacement functionality for value holders
Simplify date parsing function
Add local compiled code caching
|
|
code over a certain threshold
+ Updating Janino to 2.7.4 (from 2.6.1).
+ Added session options `exec.java_compiler`:[DEFAULT, JDK, JANINO] and `exec.java_compiler_janino_maxsize`(256k).
+ Fix a bug in FragmentExecutor.java(125) where the root operator is not stopped in case of fragment failure.
+ Print Java stack trace of the allocator code if a TLA detects that any of the ChildAllocator is not closed.
|
|
classes
+ Disallow non-static '*' imports in handwritten code.
+ Updated the current code to be in compliance.
+ Run 'rat' plugin in 'validate' phase.
|
|
|
|
ClassGenerators to support implementation of an arbitrarily complex inner class hierarchy. Introduce @RuntimeOveridden annotation to allow overrides of inner classes without requiring abstract inner classes. Update operators to use methods. Add test for inner class generation. Remove support for JDK based compilation, focusing entirely on Janino based compilation.
|
|
|
|
|
|
|
