From 15e5c14ff73b083e69f77b53bd88ced9b4ce9875 Mon Sep 17 00:00:00 2001 From: George Apaaboah <35894485+GeorgeAp@users.noreply.github.com> Date: Fri, 26 Nov 2021 15:44:13 +0100 Subject: [PATCH 1/7] [FEDE-4869] Upgrade Arrow version to 4.0.0 (#16) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Make BaseValueVector#MAX_ALLOCATION_SIZE configurable This closes #65 Some of the tests are based on the assumption that the JVM can allocate at least 2GB of memory, which is not a common occurence (JVM usually defaults at 512MB). Current Travis CI VM only have 3GB of memory total, which would have make challenging to run some of the tests on them Add a system property to change BaseValueVector.MAX_ALLOCATION_SIZE to allow to use a much smaller value during tests. * prefix arrow's version with siren * use our version of netty * updated readme about siren's changes * fixed dependency issue with our own artifactory * use our version of netty * shade the arrow memory jar * improved doc * fix readme * document the changes done to the arrow fork * ARROW-5856: [Python] [Packaging] Fix use of C++ / Cython API from wheels Author: Antoine Pitrou Closes #4884 from pitrou/ARROW-5856-cython-so-version and squashes the following commits: a411d7ad7 Avoid C++ ABI issues with DictionaryMemo eaede5be2 Revert ARROW-5082 (" Stop exporting copies of shared libraries in wheel") 4594f784e ARROW-5856: Try to fix Cython API from wheels * use our version of netty * set drill's default value * use our version of netty * bumped to 0.8.0 * update to 0.14.1 * comment unneeded modules * update release procedure with unneeded modules commented out * bump version to siren-0.14.1-1 and update readme * do not allocate a new offset buffer if the slice starts at 0 since the relative offset pointer would be unchanged * fix slice bounds * improved readme * Upgraded netty dependendcy to siren-4.1.27-3 * [FEDE-3917] netty direct memory counter deprecation with bump to siren-0.14.1-6-SNAPSHOT (#10) * Release siren-0.14.1-5 * Bump to version siren-0.14.1-6-SNAPSHOT * Use Siren Netty fork version siren-4.1.27-4 to release siren-0.14.1-6 * Bump version to release siren-0.14.1-5 using Netty Siren siren-4.1.27-4 * Set the version to siren-0.14.1-6-SNAPSHOT Co-authored-by: Martin Anseaume * Fix rebase * Fix pom * Fix rebase - Bump version to siren-4.0.0-1-SNAPSHOT * Clean-up * Comment out memory-netty module and remove use siren netty * Update siren netty version * Fix rebase * Comment netty * Fix TestValueVector * Remove unused imports * Remove unused import * Clean up * Clean up - fix checkstyle * Clean up - fix checkstyle 2 * WIP * WIP - add siren netty plugin * WIP - ignore failing test * Fix rebase as per peer review * Fix rebase python * Fix ZeroVector unused imports * Fix checkstyle * Fix rebase * Fix rebase * Fix checkstyle imports violation * Fix checkstyle and update netty version * Use Arrow's ArrowBuf instead of our own * Fix imports * Fix checkstyle * Tentative fix: workaround for the shading issue but creat new problems * fix missing import * Fix import * Fix imports * Revert ignored unit tests * Revert automatic format changes * Revert format to original format * Update vector pom as per peer review * Revert ignored TestArrowBufHasher.testHasherNegative * Clean up * Update as per peer review * Undo unnecessary change in tasks.yml * Undo unnecessary changes * Revert changes * Revert changes * Revert changes as per peer review * Update as per peer review * Add back class path dependency exclusion * Remove whitespace * Update memory access with default value * Remove class path exclusion * Update readme * Exclude memory-core from the shaded netty in memory-netty package * Add information for checking that Siren version of Netty is used * [FEDE-5144] Fix the static initialization of MemoryUtil (#17) * Remove unecessary setAccessible call * Catch all errors thrown by the setAccessible call New version of java can throw InaccessibleObjectException which Arrow didn't handle * Update netty version to the stable version siren-4.1.48-1 Co-authored-by: Laurent Goujon Co-authored-by: Stéphane Campinas Co-authored-by: Antoine Pitrou Co-authored-by: Martin Anseaume Co-authored-by: Johnny Hujol Co-authored-by: ggdupont --- README.md | 98 +++++++++++++++++++ java/adapter/orc/pom.xml | 2 +- java/format/pom.xml | 2 +- java/gandiva/pom.xml | 2 +- java/memory/memory-core/pom.xml | 2 +- .../apache/arrow/memory/BoundsChecking.java | 13 +-- .../apache/arrow/memory/util/MemoryUtil.java | 6 +- .../arrow/memory/TestBoundaryChecking.java | 4 +- java/memory/memory-netty/pom.xml | 51 +++++++++- java/memory/memory-unsafe/pom.xml | 2 +- java/memory/pom.xml | 2 +- java/pom.xml | 38 +++++-- java/vector/pom.xml | 2 +- .../src/main/codegen/includes/vv_imports.ftl | 2 + .../arrow/vector/BaseFixedWidthVector.java | 2 +- .../org/apache/arrow/vector/BitVector.java | 5 + .../apache/arrow/vector/BitVectorHelper.java | 8 +- .../apache/arrow/vector/Decimal256Vector.java | 2 +- .../apache/arrow/vector/DecimalVector.java | 2 +- .../arrow/vector/util/DecimalUtility.java | 2 +- .../arrow/vector/util/VectorAppender.java | 2 +- .../arrow/vector/TestBitVectorHelper.java | 2 +- .../apache/arrow/vector/TestValueVector.java | 2 +- 23 files changed, 211 insertions(+), 42 deletions(-) diff --git a/README.md b/README.md index 7d10b81c6e4f..c90f69801cfd 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,73 @@ under the License. --> +# Siren fork of Arrow + +- The properties `drill.enable_unsafe_memory_access` and + `arrow.enable_unsafe_memory_access` are prefixed with `siren` and their + default value is set to `true`. The first property is deprecated. + +- In order to avoid conflict with a version of `netty` used in Elasticsearch, we + relocate the netty custom package and dependency in `memory` into a package + named `siren`. The relocation is achieved thanks to the maven shade plugin. + +- The Siren's fork of `netty` is used in `vector`. This means that `netty` + imports in that module need to be prefixed with `siren`. + +## Check that Siren version of Netty is used +- In order to check that Siren version of Netty is being used, + run the unit test `CheckAccessibleTest` in + `https://github.com/sirensolutions/siren-platform/blob/master/core/src/test/java/io/siren/federate/core/common/CheckAccessibleTest.java`. +- Note: the unit test `CheckAccessibleTest` is currently ignored, please set it again to ignore after running the test. + The unit test is ignored because the settings in `CheckAccessibleTest` is not taken into account when the whole unit test suite is run, therefore it fails. + This could be because when the class is loaded, the default settings is used (which is a static block) and the new settings in the `CheckAccessibleTest` is + then not applied when the test suit is run. + +## Build + +To build the `memory`, `format` and `vector` modules: + +```sh +$ cd java +$ mvn clean package +``` + +Because of the default value change of `unsafe_memory_access` property, some +tests in `vector` fail. + +```sh +mvn -pl memory,memory/memory-core,memory/memory-netty,memory/memory-unsafe,format,vector install -Dsiren.arrow.enable_unsafe_memory_access=false -Dsiren.drill.enable_unsafe_memory_access=false +``` + +## Make a new release + +- Tests should pass. + +- Make a new version: + +```sh +mvn versions:set -DnewVersion=siren-0.14.1-2 +``` + +- tag the commit for the release + +```sh +git tag --sign siren-0.14.1-2 +```` + +- Deploy to Siren's artifactory + +```sh +$ mvn deploy -DskipTests=true -P artifactory -Dartifactory_username= -Dartifactory_password= +``` + +## Update to a new version of Apache Arrow + +- add `git@github.com:apache/arrow.git` as the `upstream` remote. +- execute `git fetch --all --tags` +- create a temporary branch from `siren-changes` +- rebase against the new tag. + # Apache Arrow [![Fuzzing Status](https://oss-fuzz-build-logs.storage.googleapis.com/badges/arrow.svg)](https://bugs.chromium.org/p/oss-fuzz/issues/list?sort=-opened&can=1&q=proj:arrow) @@ -97,6 +164,37 @@ integrations in other projects, we'd be happy to have you involved: - [Learn the format][2] - Contribute code to one of the reference implementations +### How to Contribute + +We prefer to receive contributions in the form of GitHub pull requests. Please +send pull requests against the [github.com/apache/arrow][4] repository. + +If you are looking for some ideas on what to contribute, check out the [JIRA +issues][3] for the Apache Arrow project. Comment on the issue and/or contact +[dev@arrow.apache.org](http://mail-archives.apache.org/mod_mbox/arrow-dev/) +with your questions and ideas. + +If you’d like to report a bug but don’t have time to fix it, you can still post +it on JIRA, or email the mailing list +[dev@arrow.apache.org](http://mail-archives.apache.org/mod_mbox/arrow-dev/) + +To contribute a patch: + +1. Break your work into small, single-purpose patches if possible. It’s much +harder to merge in a large change with a lot of disjoint features. +2. Create a JIRA for your patch on the [Arrow Project +JIRA](https://issues.apache.org/jira/browse/ARROW). +3. Submit the patch as a GitHub pull request against the master branch. For a +tutorial, see the GitHub guides on forking a repo and sending a pull +request. Prefix your pull request name with the JIRA name (ex: +https://github.com/apache/arrow/pull/240). +4. Make sure that your code passes the unit tests. You can find instructions +how to run the unit tests for each Arrow component in its respective README +file. +5. Add new unit tests for your code. + +Thank you in advance for your contributions! + [1]: mailto:dev-subscribe@arrow.apache.org [2]: https://github.com/apache/arrow/tree/master/format [3]: https://issues.apache.org/jira/browse/ARROW diff --git a/java/adapter/orc/pom.xml b/java/adapter/orc/pom.xml index 3e626c7fd6ae..de7fd8a8fee3 100644 --- a/java/adapter/orc/pom.xml +++ b/java/adapter/orc/pom.xml @@ -87,7 +87,7 @@ org.apache.arrow arrow-java-root - 7.0.0 + siren-7.0.0-1-SNAPSHOT ../../pom.xml diff --git a/java/format/pom.xml b/java/format/pom.xml index 6214bfdecd39..d905ae662174 100644 --- a/java/format/pom.xml +++ b/java/format/pom.xml @@ -15,7 +15,7 @@ arrow-java-root org.apache.arrow - 7.0.0 + siren-7.0.0-1-SNAPSHOT arrow-format diff --git a/java/gandiva/pom.xml b/java/gandiva/pom.xml index f02f969e710b..46194c93777e 100644 --- a/java/gandiva/pom.xml +++ b/java/gandiva/pom.xml @@ -14,7 +14,7 @@ org.apache.arrow arrow-java-root - 7.0.0 + siren-7.0.0-1-SNAPSHOT org.apache.arrow.gandiva diff --git a/java/memory/memory-core/pom.xml b/java/memory/memory-core/pom.xml index 40be427a9e89..02a76475ec85 100644 --- a/java/memory/memory-core/pom.xml +++ b/java/memory/memory-core/pom.xml @@ -13,7 +13,7 @@ arrow-memory org.apache.arrow - 7.0.0 + siren-7.0.0-1-SNAPSHOT 4.0.0 diff --git a/java/memory/memory-core/src/main/java/org/apache/arrow/memory/BoundsChecking.java b/java/memory/memory-core/src/main/java/org/apache/arrow/memory/BoundsChecking.java index bbf7ff34dc96..a6d798bd093e 100644 --- a/java/memory/memory-core/src/main/java/org/apache/arrow/memory/BoundsChecking.java +++ b/java/memory/memory-core/src/main/java/org/apache/arrow/memory/BoundsChecking.java @@ -34,14 +34,15 @@ public class BoundsChecking { static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(BoundsChecking.class); static { - String envProperty = System.getenv("ARROW_ENABLE_UNSAFE_MEMORY_ACCESS"); - String oldProperty = System.getProperty("drill.enable_unsafe_memory_access"); + String envProperty = System.getenv().getOrDefault("SIREN_ARROW_ENABLE_UNSAFE_MEMORY_ACCESS", "true"); + String oldProperty = System.getProperty("siren.drill.enable_unsafe_memory_access", "true"); if (oldProperty != null) { - logger.warn("\"drill.enable_unsafe_memory_access\" has been renamed to \"arrow.enable_unsafe_memory_access\""); - logger.warn("\"arrow.enable_unsafe_memory_access\" can be set to: " + - " true (to not check) or false (to check, default)"); + logger.warn("\"siren.drill.enable_unsafe_memory_access\" has been renamed to " + + "\"siren.arrow.enable_unsafe_memory_access\""); + logger.warn("\"siren.arrow.enable_unsafe_memory_access\" can be set to: " + + " true (to not check, default) or false (to check)"); } - String newProperty = System.getProperty("arrow.enable_unsafe_memory_access"); + String newProperty = System.getProperty("siren.arrow.enable_unsafe_memory_access", "true"); // The priority of determining the unsafe flag: // 1. The system properties take precedence over the environmental variable. diff --git a/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java b/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java index 16ef39702ca3..ce7a4bd5e210 100644 --- a/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java +++ b/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java @@ -81,7 +81,6 @@ public Object run() { // get the offset of the address field in a java.nio.Buffer object Field addressField = java.nio.Buffer.class.getDeclaredField("address"); - addressField.setAccessible(true); BYTE_BUFFER_ADDRESS_OFFSET = UNSAFE.objectFieldOffset(addressField); Constructor directBufferConstructor; @@ -99,10 +98,7 @@ public Object run() { constructor.setAccessible(true); logger.debug("Constructor for direct buffer found and made accessible"); return constructor; - } catch (NoSuchMethodException e) { - logger.debug("Cannot get constructor for direct buffer allocation", e); - return e; - } catch (SecurityException e) { + } catch (Exception e) { logger.debug("Cannot get constructor for direct buffer allocation", e); return e; } diff --git a/java/memory/memory-core/src/test/java/org/apache/arrow/memory/TestBoundaryChecking.java b/java/memory/memory-core/src/test/java/org/apache/arrow/memory/TestBoundaryChecking.java index 5b86bed406b3..2e423119a54d 100644 --- a/java/memory/memory-core/src/test/java/org/apache/arrow/memory/TestBoundaryChecking.java +++ b/java/memory/memory-core/src/test/java/org/apache/arrow/memory/TestBoundaryChecking.java @@ -55,7 +55,7 @@ private boolean getFlagValue(ClassLoader classLoader) throws Exception { } /** - * Ensure the flag for bounds checking is enabled by default. + * Siren: Ensure the flag for bounds checking is disabled by default. * This will protect users from JVM crashes. */ @Test @@ -63,7 +63,7 @@ public void testDefaultValue() throws Exception { ClassLoader classLoader = copyClassLoader(); if (classLoader != null) { boolean boundsCheckingEnabled = getFlagValue(classLoader); - Assert.assertTrue(boundsCheckingEnabled); + Assert.assertFalse(boundsCheckingEnabled); } } diff --git a/java/memory/memory-netty/pom.xml b/java/memory/memory-netty/pom.xml index 16af33bbea76..3524eeacfb8e 100644 --- a/java/memory/memory-netty/pom.xml +++ b/java/memory/memory-netty/pom.xml @@ -13,7 +13,7 @@ arrow-memory org.apache.arrow - 7.0.0 + siren-7.0.0-1-SNAPSHOT 4.0.0 @@ -35,9 +35,11 @@ io.netty netty-common + org.slf4j slf4j-api + provided org.immutables @@ -45,6 +47,53 @@ + + + + org.apache.maven.plugins + maven-shade-plugin + 3.1.0 + + + package + + shade + + + false + ${project.build.directory}/dependency-reduced-pom.xml + + + io.netty + siren.io.netty + + + + + org.slf4j + com.google.code.findbugs + com.google.guava + org.apache.arrow:arrow-memory-core:*:* + + + + + + *:* + + META-INF/*.SF + META-INF/*.DSA + META-INF/*.RSA + + + + + + + + + + + + + + + + + @@ -846,6 +846,24 @@ + + + artifactory + + + + artifactory-releases + artifactory-releases + ${artifactory.url}/libs-release-local + + + artifactory-snapshots + artifactory-snapshots + ${artifactory.url}/libs-snapshot-local + + + + diff --git a/java/vector/pom.xml b/java/vector/pom.xml index da6b5c3f9b52..dc82eb336940 100644 --- a/java/vector/pom.xml +++ b/java/vector/pom.xml @@ -14,7 +14,7 @@ org.apache.arrow arrow-java-root - 7.0.0 + siren-7.0.0-1-SNAPSHOT arrow-vector Arrow Vectors diff --git a/java/vector/src/main/codegen/includes/vv_imports.ftl b/java/vector/src/main/codegen/includes/vv_imports.ftl index c9a8820b258b..cd0e2d7e2a4c 100644 --- a/java/vector/src/main/codegen/includes/vv_imports.ftl +++ b/java/vector/src/main/codegen/includes/vv_imports.ftl @@ -20,6 +20,8 @@ import static org.apache.arrow.util.Preconditions.checkState; import com.google.flatbuffers.FlatBufferBuilder; +import siren.io.netty.buffer.*; + import org.apache.arrow.memory.*; import org.apache.arrow.util.Preconditions; import org.apache.arrow.vector.types.Types; diff --git a/java/vector/src/main/java/org/apache/arrow/vector/BaseFixedWidthVector.java b/java/vector/src/main/java/org/apache/arrow/vector/BaseFixedWidthVector.java index ded58b22b00c..70b7c0d7a432 100644 --- a/java/vector/src/main/java/org/apache/arrow/vector/BaseFixedWidthVector.java +++ b/java/vector/src/main/java/org/apache/arrow/vector/BaseFixedWidthVector.java @@ -37,7 +37,7 @@ import org.apache.arrow.vector.util.OversizedAllocationException; import org.apache.arrow.vector.util.TransferPair; -import io.netty.util.internal.PlatformDependent; +import siren.io.netty.util.internal.PlatformDependent; /** * BaseFixedWidthVector provides an abstract interface for diff --git a/java/vector/src/main/java/org/apache/arrow/vector/BitVector.java b/java/vector/src/main/java/org/apache/arrow/vector/BitVector.java index 3bcfd983e641..1935b45f7bd9 100644 --- a/java/vector/src/main/java/org/apache/arrow/vector/BitVector.java +++ b/java/vector/src/main/java/org/apache/arrow/vector/BitVector.java @@ -119,6 +119,11 @@ public void setInitialCapacity(int valueCount) { lastValueCapacity = valueCount; } + /** + * Get the current value capacity for the vector. + * + * @return number of elements that vector can hold. + */ @Override protected int getValueBufferValueCapacity() { return capAtMaxInt(valueBuffer.capacity() * 8); diff --git a/java/vector/src/main/java/org/apache/arrow/vector/BitVectorHelper.java b/java/vector/src/main/java/org/apache/arrow/vector/BitVectorHelper.java index c2c88f2ba378..53f84a884277 100644 --- a/java/vector/src/main/java/org/apache/arrow/vector/BitVectorHelper.java +++ b/java/vector/src/main/java/org/apache/arrow/vector/BitVectorHelper.java @@ -17,10 +17,10 @@ package org.apache.arrow.vector; -import static io.netty.util.internal.PlatformDependent.getByte; -import static io.netty.util.internal.PlatformDependent.getInt; -import static io.netty.util.internal.PlatformDependent.getLong; import static org.apache.arrow.memory.util.LargeMemoryUtil.checkedCastToInt; +import static siren.io.netty.util.internal.PlatformDependent.getByte; +import static siren.io.netty.util.internal.PlatformDependent.getInt; +import static siren.io.netty.util.internal.PlatformDependent.getLong; import org.apache.arrow.memory.ArrowBuf; import org.apache.arrow.memory.BoundsChecking; @@ -28,7 +28,7 @@ import org.apache.arrow.vector.ipc.message.ArrowFieldNode; import org.apache.arrow.vector.util.DataSizeRoundingUtil; -import io.netty.util.internal.PlatformDependent; +import siren.io.netty.util.internal.PlatformDependent; /** * Helper class for performing generic operations on a bit vector buffer. diff --git a/java/vector/src/main/java/org/apache/arrow/vector/Decimal256Vector.java b/java/vector/src/main/java/org/apache/arrow/vector/Decimal256Vector.java index c5fef82d0527..550119200da6 100644 --- a/java/vector/src/main/java/org/apache/arrow/vector/Decimal256Vector.java +++ b/java/vector/src/main/java/org/apache/arrow/vector/Decimal256Vector.java @@ -35,7 +35,7 @@ import org.apache.arrow.vector.util.DecimalUtility; import org.apache.arrow.vector.util.TransferPair; -import io.netty.util.internal.PlatformDependent; +import siren.io.netty.util.internal.PlatformDependent; /** * Decimal256Vector implements a fixed width vector (32 bytes) of diff --git a/java/vector/src/main/java/org/apache/arrow/vector/DecimalVector.java b/java/vector/src/main/java/org/apache/arrow/vector/DecimalVector.java index f988f4f94902..f053f764de43 100644 --- a/java/vector/src/main/java/org/apache/arrow/vector/DecimalVector.java +++ b/java/vector/src/main/java/org/apache/arrow/vector/DecimalVector.java @@ -35,7 +35,7 @@ import org.apache.arrow.vector.util.DecimalUtility; import org.apache.arrow.vector.util.TransferPair; -import io.netty.util.internal.PlatformDependent; +import siren.io.netty.util.internal.PlatformDependent; /** * DecimalVector implements a fixed width vector (16 bytes) of diff --git a/java/vector/src/main/java/org/apache/arrow/vector/util/DecimalUtility.java b/java/vector/src/main/java/org/apache/arrow/vector/util/DecimalUtility.java index f778bcb209fa..692714487fbf 100644 --- a/java/vector/src/main/java/org/apache/arrow/vector/util/DecimalUtility.java +++ b/java/vector/src/main/java/org/apache/arrow/vector/util/DecimalUtility.java @@ -24,7 +24,7 @@ import org.apache.arrow.memory.ArrowBuf; -import io.netty.util.internal.PlatformDependent; +import siren.io.netty.util.internal.PlatformDependent; /** * Utility methods for configurable precision Decimal values (e.g. {@link BigDecimal}). diff --git a/java/vector/src/main/java/org/apache/arrow/vector/util/VectorAppender.java b/java/vector/src/main/java/org/apache/arrow/vector/util/VectorAppender.java index ea78917c3ddb..003a8c97b611 100644 --- a/java/vector/src/main/java/org/apache/arrow/vector/util/VectorAppender.java +++ b/java/vector/src/main/java/org/apache/arrow/vector/util/VectorAppender.java @@ -39,7 +39,7 @@ import org.apache.arrow.vector.complex.NonNullableStructVector; import org.apache.arrow.vector.complex.UnionVector; -import io.netty.util.internal.PlatformDependent; +import siren.io.netty.util.internal.PlatformDependent; /** * Utility to append two vectors together. diff --git a/java/vector/src/test/java/org/apache/arrow/vector/TestBitVectorHelper.java b/java/vector/src/test/java/org/apache/arrow/vector/TestBitVectorHelper.java index 4563577202bb..57f64bbde58e 100644 --- a/java/vector/src/test/java/org/apache/arrow/vector/TestBitVectorHelper.java +++ b/java/vector/src/test/java/org/apache/arrow/vector/TestBitVectorHelper.java @@ -27,7 +27,7 @@ import org.apache.arrow.vector.ipc.message.ArrowFieldNode; import org.junit.Test; -import io.netty.util.internal.PlatformDependent; +import siren.io.netty.util.internal.PlatformDependent; public class TestBitVectorHelper { @Test diff --git a/java/vector/src/test/java/org/apache/arrow/vector/TestValueVector.java b/java/vector/src/test/java/org/apache/arrow/vector/TestValueVector.java index 8858281063f4..21da52ceb3f1 100644 --- a/java/vector/src/test/java/org/apache/arrow/vector/TestValueVector.java +++ b/java/vector/src/test/java/org/apache/arrow/vector/TestValueVector.java @@ -1811,7 +1811,7 @@ public void testSetLastSetUsage() { assertEquals(40, vector.offsetBuffer.getInt(17 * BaseVariableWidthVector.OFFSET_WIDTH)); assertEquals(40, vector.offsetBuffer.getInt(18 * BaseVariableWidthVector.OFFSET_WIDTH)); assertEquals(40, vector.offsetBuffer.getInt(19 * BaseVariableWidthVector.OFFSET_WIDTH)); - + vector.set(19, STR6); assertArrayEquals(STR6, vector.get(19)); assertEquals(40, vector.offsetBuffer.getInt(19 * BaseVariableWidthVector.OFFSET_WIDTH)); From c2b620c085653aa5051de00248a92034348fb80b Mon Sep 17 00:00:00 2001 From: George Apaaboah Date: Fri, 26 Nov 2021 16:02:17 +0100 Subject: [PATCH 2/7] Bumped version number to siren-4.0.0-1 --- java/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/java/pom.xml b/java/pom.xml index f856676abc15..37ad29df416b 100644 --- a/java/pom.xml +++ b/java/pom.xml @@ -862,7 +862,7 @@ ${artifactory.url}/libs-snapshot-local - + From 49b9a17801c59f4c2602d0c2302530bb7d620ce5 Mon Sep 17 00:00:00 2001 From: George Apaaboah Date: Mon, 28 Mar 2022 10:22:21 +0200 Subject: [PATCH 3/7] Fix rebase --- java/vector/pom.xml | 6 ------ 1 file changed, 6 deletions(-) diff --git a/java/vector/pom.xml b/java/vector/pom.xml index dc82eb336940..72baafe8b260 100644 --- a/java/vector/pom.xml +++ b/java/vector/pom.xml @@ -121,12 +121,6 @@ test - - - org.apache.arrow:arrow-memory-netty - - netty - From cc2017e35076bc627fc6d4a0f65c7c34f75f8c41 Mon Sep 17 00:00:00 2001 From: George Apaaboah Date: Tue, 12 Apr 2022 09:37:11 +0200 Subject: [PATCH 4/7] Remove scope from arrow-memory-netty to make netty available for entire project else our shading will fail. Signed-off-by: George Apaaboah --- java/vector/pom.xml | 1 - 1 file changed, 1 deletion(-) diff --git a/java/vector/pom.xml b/java/vector/pom.xml index 72baafe8b260..f7b2d27a9de5 100644 --- a/java/vector/pom.xml +++ b/java/vector/pom.xml @@ -53,7 +53,6 @@ org.apache.arrow arrow-memory-netty ${project.version} - test org.apache.arrow From ae76e23ff4e84ede7fb251f09cf49675baf7592d Mon Sep 17 00:00:00 2001 From: George Apaaboah Date: Tue, 12 Apr 2022 09:59:05 +0200 Subject: [PATCH 5/7] Fix readme - remove duplicate section on how to contribute --- README.md | 31 ------------------------------- 1 file changed, 31 deletions(-) diff --git a/README.md b/README.md index c90f69801cfd..8c714d38f537 100644 --- a/README.md +++ b/README.md @@ -164,37 +164,6 @@ integrations in other projects, we'd be happy to have you involved: - [Learn the format][2] - Contribute code to one of the reference implementations -### How to Contribute - -We prefer to receive contributions in the form of GitHub pull requests. Please -send pull requests against the [github.com/apache/arrow][4] repository. - -If you are looking for some ideas on what to contribute, check out the [JIRA -issues][3] for the Apache Arrow project. Comment on the issue and/or contact -[dev@arrow.apache.org](http://mail-archives.apache.org/mod_mbox/arrow-dev/) -with your questions and ideas. - -If you’d like to report a bug but don’t have time to fix it, you can still post -it on JIRA, or email the mailing list -[dev@arrow.apache.org](http://mail-archives.apache.org/mod_mbox/arrow-dev/) - -To contribute a patch: - -1. Break your work into small, single-purpose patches if possible. It’s much -harder to merge in a large change with a lot of disjoint features. -2. Create a JIRA for your patch on the [Arrow Project -JIRA](https://issues.apache.org/jira/browse/ARROW). -3. Submit the patch as a GitHub pull request against the master branch. For a -tutorial, see the GitHub guides on forking a repo and sending a pull -request. Prefix your pull request name with the JIRA name (ex: -https://github.com/apache/arrow/pull/240). -4. Make sure that your code passes the unit tests. You can find instructions -how to run the unit tests for each Arrow component in its respective README -file. -5. Add new unit tests for your code. - -Thank you in advance for your contributions! - [1]: mailto:dev-subscribe@arrow.apache.org [2]: https://github.com/apache/arrow/tree/master/format [3]: https://issues.apache.org/jira/browse/ARROW From dd462540b1249697a9814ed1ca9217ab231560d3 Mon Sep 17 00:00:00 2001 From: George Apaaboah Date: Thu, 14 Apr 2022 15:20:03 +0200 Subject: [PATCH 6/7] Update readme as per review --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 8c714d38f537..f6a9073415f9 100644 --- a/README.md +++ b/README.md @@ -55,7 +55,7 @@ tests in `vector` fail. mvn -pl memory,memory/memory-core,memory/memory-netty,memory/memory-unsafe,format,vector install -Dsiren.arrow.enable_unsafe_memory_access=false -Dsiren.drill.enable_unsafe_memory_access=false ``` -## Make a new release +## Make a new release of Siren's Apache Arrow - Tests should pass. @@ -77,7 +77,7 @@ git tag --sign siren-0.14.1-2 $ mvn deploy -DskipTests=true -P artifactory -Dartifactory_username= -Dartifactory_password= ``` -## Update to a new version of Apache Arrow +## Update to a new version of Siren's Apache Arrow - add `git@github.com:apache/arrow.git` as the `upstream` remote. - execute `git fetch --all --tags` From 33d038b520efadc9f5ba5047e4a89b78d2848eee Mon Sep 17 00:00:00 2001 From: George Apaaboah Date: Wed, 11 May 2022 15:07:13 +0200 Subject: [PATCH 7/7] Use stable netty version siren-4.1.68-1 after the release of netty Signed-off-by: George Apaaboah --- java/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/java/pom.xml b/java/pom.xml index 37ad29df416b..12ee7d0316d9 100644 --- a/java/pom.xml +++ b/java/pom.xml @@ -33,7 +33,7 @@ 5.4.0 1.7.25 30.1.1-jre - siren-4.1.68-1-SNAPSHOT + siren-4.1.68-1 2.11.4 2.7.1 1.12.0