working on more data structures

LICENSE.txt

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

build.gradle

@@ -28,6 +28,7 @@ subprojects {
     apply from: rootProject.file('gradle/ide/idea.gradle')
     apply from: rootProject.file('gradle/compile/java.gradle')
     apply from: rootProject.file('gradle/test/junit5.gradle')
+    apply from: rootProject.file('gradle/test/jmh.gradle')
     apply from: rootProject.file('gradle/publishing/publication.gradle')
 }
 

datastructures-bytes/build.gradle

@@ -0,0 +1,6 @@
+dependencies {
+    api "net.openhft:chronicle-core:2.21ea14"
+    testImplementation "org.junit.vintage:junit-vintage-engine:5.7.0"
+    testImplementation "junit:junit:4.13"
+    testImplementation "net.openhft:affinity:3.21ea0"
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/AbstractBytesStore.java

@@ -0,0 +1,72 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ *  Licensed under the Apache License, Version 2.0 (the "License");
+ *  you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.bytes.algo.BytesStoreHash;
+import net.openhft.chronicle.core.io.AbstractReferenceCounted;
+
+import java.nio.BufferUnderflowException;
+
+public abstract class AbstractBytesStore<B extends BytesStore<B, Underlying>, Underlying>
+        extends AbstractReferenceCounted
+        implements BytesStore<B, Underlying> {
+
+    protected AbstractBytesStore() {
+    }
+
+    protected AbstractBytesStore(boolean monitored) {
+        super(monitored);
+    }
+
+    @Override
+    public int peekUnsignedByte(long offset) throws BufferUnderflowException {
+        return offset >= readLimit() ? -1 : readUnsignedByte(offset);
+    }
+
+    @Override
+    public int hashCode() {
+        return BytesStoreHash.hash32(this);
+    }
+
+    @Override
+    public long readPosition() {
+        return 0L;
+    }
+
+    @Override
+    public long readRemaining() {
+        return readLimit() - readPosition();
+    }
+
+    @Override
+    public long writeRemaining() {
+        return writeLimit() - writePosition();
+    }
+
+    @Override
+    public long start() {
+        return 0L;
+    }
+
+    @Override
+    protected boolean performReleaseInBackground() {
+        return isDirectMemory();
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/AppendableUtil.java

@@ -0,0 +1,332 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.Maths;
+import net.openhft.chronicle.core.annotation.ForceInline;
+import net.openhft.chronicle.core.annotation.Java9;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+import java.io.IOException;
+import java.io.UTFDataFormatException;
+import java.nio.BufferOverflowException;
+import java.nio.BufferUnderflowException;
+
+@SuppressWarnings("rawtypes")
+public enum AppendableUtil {
+    ;
+
+    public static void setCharAt( Appendable sb, int index, char ch)
+            throws IllegalArgumentException, BufferOverflowException {
+        if (sb instanceof StringBuilder)
+            ((StringBuilder) sb).setCharAt(index, ch);
+        else if (sb instanceof Bytes)
+            ((Bytes) sb).writeByte(index, ch);
+        else
+            throw new IllegalArgumentException("" + sb.getClass());
+    }
+
+    public static void parseUtf8( BytesStore bs, StringBuilder sb, boolean utf, int length) throws UTFDataFormatRuntimeException {
+        BytesInternal.parseUtf8(bs, bs.readPosition(), sb, utf, length);
+    }
+
+    @ForceInline
+    public static void setLength( Appendable sb, int newLength)
+            throws BufferUnderflowException, IllegalArgumentException {
+        if (sb instanceof StringBuilder)
+            ((StringBuilder) sb).setLength(newLength);
+        else if (sb instanceof Bytes)
+            ((Bytes) sb).readPositionRemaining(0, newLength);
+        else
+            throw new IllegalArgumentException("" + sb.getClass());
+    }
+
+    public static void append( Appendable sb, double value)
+            throws IllegalArgumentException, BufferOverflowException {
+        if (sb instanceof StringBuilder)
+            ((StringBuilder) sb).append(value);
+        else if (sb instanceof Bytes)
+            ((Bytes) sb).append(value);
+        else
+            throw new IllegalArgumentException("" + sb.getClass());
+    }
+
+    public static void append( Appendable sb, long value)
+            throws IllegalArgumentException, BufferOverflowException {
+        if (sb instanceof StringBuilder)
+            ((StringBuilder) sb).append(value);
+        else if (sb instanceof Bytes)
+            ((Bytes) sb).append(value);
+        else
+            throw new IllegalArgumentException("" + sb.getClass());
+    }
+
+    public static <ACS extends Appendable & CharSequence> void append( ACS sb, String str) {
+        try {
+            sb.append(str);
+        } catch (IOException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+    public static void read8bitAndAppend( StreamingDataInput bytes,
+                                          StringBuilder appendable,
+                                          StopCharsTester tester) {
+        while (true) {
+            int c = bytes.readUnsignedByte();
+            if (tester.isStopChar(c, bytes.peekUnsignedByte()))
+                return;
+            appendable.append((char) c);
+            if (bytes.readRemaining() == 0)
+                return;
+        }
+    }
+
+    public static void readUTFAndAppend( StreamingDataInput bytes,
+                                         Appendable appendable,
+                                         StopCharsTester tester)
+            throws BufferUnderflowException {
+        try {
+            readUtf8AndAppend(bytes, appendable, tester);
+        } catch (IOException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+    public static void readUtf8AndAppend( StreamingDataInput bytes,
+                                          Appendable appendable,
+                                          StopCharsTester tester)
+            throws BufferUnderflowException, IOException {
+        while (true) {
+            int c = bytes.readUnsignedByte();
+            if (c >= 128) {
+                bytes.readSkip(-1);
+                break;
+            }
+            // this is used for array class such as !type byte[]
+            if (c == '[' && bytes.peekUnsignedByte() == ']') {
+                appendable.append((char) c);
+                appendable.append((char) bytes.readUnsignedByte());
+                if (bytes.readRemaining() == 0)
+                    return;
+                continue;
+            }
+
+            if (tester.isStopChar(c, bytes.peekUnsignedByte()))
+                return;
+            appendable.append((char) c);
+            if (bytes.readRemaining() == 0)
+                return;
+        }
+
+        for (int c; (c = bytes.readUnsignedByte()) >= 0; ) {
+            switch (c >> 4) {
+                case 0:
+                case 1:
+                case 2:
+                case 3:
+                case 4:
+                case 5:
+                case 6:
+                case 7:
+                    /* 0xxxxxxx */
+                    if (tester.isStopChar(c, bytes.peekUnsignedByte()))
+                        return;
+                    appendable.append((char) c);
+                    break;
+
+                case 12:
+                case 13: {
+                    /* 110x xxxx 10xx xxxx */
+                    int char2 = bytes.readUnsignedByte();
+                    if ((char2 & 0xC0) != 0x80)
+                        throw new UTFDataFormatException(
+                                "malformed input around byte " + Integer.toHexString(char2));
+                    int c2 = (char) (((c & 0x1F) << 6) |
+                            (char2 & 0x3F));
+                    if (tester.isStopChar(c2, bytes.peekUnsignedByte()))
+                        return;
+                    appendable.append((char) c2);
+                    break;
+                }
+
+                case 14: {
+                    /* 1110 xxxx 10xx xxxx 10xx xxxx */
+                    int char2 = bytes.readUnsignedByte();
+                    int char3 = bytes.readUnsignedByte();
+
+                    if (((char2 & 0xC0) != 0x80))
+                        throw new UTFDataFormatException(
+                                "malformed input around byte " + Integer.toHexString(char2));
+                    if ((char3 & 0xC0) != 0x80)
+                        throw new UTFDataFormatException(
+                                "malformed input around byte " + Integer.toHexString(char3));
+                    int c3 = (char) (((c & 0x0F) << 12) |
+                            ((char2 & 0x3F) << 6) |
+                            (char3 & 0x3F));
+                    if (tester.isStopChar(c3, bytes.peekUnsignedByte()))
+                        return;
+                    appendable.append((char) c3);
+                    break;
+                }
+
+                default:
+                    /* 10xx xxxx, 1111 xxxx */
+                    throw new UTFDataFormatException(
+                            "malformed input around byte " + Integer.toHexString(c));
+            }
+        }
+    }
+
+    public static void parse8bit_SB1( Bytes bytes,  StringBuilder sb, int length)
+            throws BufferUnderflowException {
+        if (length > bytes.readRemaining())
+            throw new BufferUnderflowException();
+         NativeBytesStore nbs = (NativeBytesStore) bytes.bytesStore();
+        long offset = bytes.readPosition();
+        int count = BytesInternal.parse8bit_SB1(offset, nbs, sb, length);
+        bytes.readSkip(count);
+    }
+
+    public static void parse8bit( StreamingDataInput bytes, Appendable appendable, int utflen)
+            throws BufferUnderflowException, IOException {
+        if (appendable instanceof StringBuilder) {
+             final StringBuilder sb = (StringBuilder) appendable;
+            if (bytes instanceof Bytes && ((Bytes) bytes).bytesStore() instanceof NativeBytesStore) {
+                parse8bit_SB1((Bytes) bytes, sb, utflen);
+            } else {
+                BytesInternal.parse8bit1(bytes, sb, utflen);
+            }
+        } else {
+            BytesInternal.parse8bit1(bytes, appendable, utflen);
+        }
+    }
+
+    public static <ACS extends Appendable & CharSequence> void append(ACS a, CharSequence cs, long start, long len) {
+        if (a instanceof StringBuilder) {
+            if (cs instanceof Bytes)
+                ((StringBuilder) a).append(Bytes.toString(((Bytes) cs), start, len));
+            else
+                ((StringBuilder) a).append(cs.subSequence(Maths.toInt32(start), Maths.toInt32(len)));
+        } else if (a instanceof Bytes) {
+            ((Bytes) a).appendUtf8(cs, Maths.toInt32(start), Maths.toInt32(len));
+        } else {
+            throw new UnsupportedOperationException();
+        }
+    }
+
+    public static long findUtf8Length( CharSequence str) throws IndexOutOfBoundsException {
+        int strlen = str.length();
+        long utflen = strlen;/* use charAt instead of copying String to char array */
+        for (int i = 0; i < strlen; i++) {
+            char c = str.charAt(i);
+            if (c <= 0x007F) {
+                continue;
+            }
+            if (c <= 0x07FF) {
+                utflen++;
+
+            } else {
+                utflen += 2;
+            }
+        }
+        return utflen;
+    }
+
+    @Java9
+    public static long findUtf8Length( byte[] bytes, byte coder) {
+        long utflen;
+
+        if (coder == 0) {
+            int strlen = bytes.length;
+            utflen = bytes.length;
+
+            //noinspection ForLoopReplaceableByForEach
+            for (int i = 0; i < strlen; i++) {
+                int b = (bytes[i] & 0xFF);
+
+                if (b > 0x007F) {
+                    utflen++;
+                }
+            }
+        } else {
+            int strlen = bytes.length;
+            utflen = 0;/* use charAt instead of copying String to char array */
+            for (int i = 0; i < strlen; i += 2) {
+                char c = (char) (((bytes[i + 1] & 0xFF) << 8) | (bytes[i] & 0xFF));
+
+                if (c <= 0x007F) {
+                    utflen += 1;
+                    continue;
+                }
+                if (c <= 0x07FF) {
+                    utflen += 2;
+                } else {
+                    utflen += 3;
+                }
+            }
+        }
+
+        return utflen;
+    }
+
+    @Java9
+    public static long findUtf8Length( byte[] chars) {
+        long utflen = 0; /* use charAt instead of copying String to char array */
+        int strlen = chars.length;
+        for (int i = 0; i < strlen; i++) {
+            int c = chars[i] & 0xFF; // unsigned byte
+
+            if (c == 0) { // we have hit end of string
+                break;
+            }
+
+            if (c >= 0xF0) {
+                utflen += 4;
+                i += 3;
+            } else if (c >= 0xE0) {
+                utflen += 3;
+                i += 2;
+            } else if (c >= 0xC0) {
+                utflen += 2;
+                i += 1;
+            } else {
+                utflen += 1;
+            }
+        }
+        return utflen;
+    }
+
+    public static long findUtf8Length( char[] chars) {
+        long utflen = chars.length;/* use charAt instead of copying String to char array */
+        for (char c : chars) {
+            if (c <= 0x007F) {
+                continue;
+            }
+            if (c <= 0x07FF) {
+                utflen++;
+
+            } else {
+                utflen += 2;
+            }
+        }
+        return utflen;
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/BinaryBytesMethodWriterInvocationHandler.java

@@ -0,0 +1,55 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.Jvm;
+import net.openhft.chronicle.core.util.AbstractInvocationHandler;
+
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.function.Function;
+
+public class BinaryBytesMethodWriterInvocationHandler extends AbstractInvocationHandler implements BytesMethodWriterInvocationHandler {
+    private final Function<Method, MethodEncoder> methodToId;
+    @SuppressWarnings("rawtypes")
+    private final BytesOut out;
+    private final Map<Method, MethodEncoder> methodToIdMap = new LinkedHashMap<>();
+
+    @SuppressWarnings("rawtypes")
+    public BinaryBytesMethodWriterInvocationHandler(Function<Method, MethodEncoder> methodToId, BytesOut out) {
+        super(HashMap::new);
+        this.methodToId = methodToId;
+        this.out = out;
+    }
+
+    @Override
+    protected Object doInvoke(Object proxy, Method method, Object[] args) throws InvocationTargetException, IllegalAccessException {
+        MethodEncoder info = methodToIdMap.computeIfAbsent(method, methodToId);
+        if (info == null) {
+            Jvm.warn().on(getClass(), "Unknown method " + method + " ignored");
+            return null;
+        }
+        out.comment(method.getName());
+        out.writeStopBit(info.messageId());
+        info.encode(args, out);
+        return null;
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/BinaryWireCode.java

@@ -0,0 +1,126 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+import org.jetbrains.annotations.NotNull;
+
+import java.lang.reflect.Field;
+
+public interface BinaryWireCode {
+
+    // sequence of length 0 - 255 bytes
+    int BYTES_LENGTH8 = 0x80;
+    //     sequence of length 0 - 2^16-1 bytes
+    int BYTES_LENGTH16 = 0x81;
+    // sequence of length 0 - 2^32-1
+    int BYTES_LENGTH32 = 0x82;
+    // sequence of length 0 - 255
+//        public static final int BYTES_LENGTH64 = 0x83;
+
+    int FIELD_ANCHOR = 0x87;
+    int ANCHOR = 0x88;
+    int UPDATED_ALIAS = 0x89;
+
+    // an array of unsigned bytes
+    int U8_ARRAY = 0x8A;
+    //        public static final int U16_ARRAY = 0x8B;
+//        public static final int I32_ARRAY = 0x8C;
+    int I64_ARRAY = 0x8D;
+    int PADDING32 = 0x8E;
+    int PADDING = 0x8F;
+
+    int FLOAT32 = 0x90;
+    int FLOAT64 = 0x91;
+    int FLOAT_STOP_2 = 0x92;
+    int FLOAT_STOP_4 = 0x94;
+    int FLOAT_STOP_6 = 0x96;
+    int FLOAT_SET_LOW_0 = 0x9A;
+    int FLOAT_SET_LOW_2 = 0x9B;
+    int FLOAT_SET_LOW_4 = 0x9C;
+    // 0x98 - 0x9F
+
+    int UUID = 0xA0;
+    int UINT8 = 0xA1;
+    int UINT16 = 0xA2;
+    int UINT32 = 0xA3;
+    int INT8 = 0xA4;
+    int INT16 = 0xA5;
+    int INT32 = 0xA6;
+    int INT64 = 0xA7;
+    int SET_LOW_INT8 = 0xA8;
+    int SET_LOW_INT16 = 0xA9;
+    //    public static final int FIXED_5 = 0xAA;
+//    public static final int FIXED_4 = 0xAB;
+//    public static final int FIXED_3 = 0xAC;
+//    public static final int FIXED_2 = 0xAD;
+    int STOP_BIT = 0xAE;
+    int INT64_0x = 0xAF;
+
+    int FALSE = 0xB0;
+    int TRUE = 0xB1;
+    int TIME = 0xB2;
+    int DATE = 0xB3;
+    int DATE_TIME = 0xB4;
+    int ZONED_DATE_TIME = 0xB5;
+    int TYPE_PREFIX = 0xB6;
+    int FIELD_NAME_ANY = 0xB7;
+    int STRING_ANY = 0xB8;
+    int EVENT_NAME = 0xB9;
+    int FIELD_NUMBER = 0xBA;
+    int NULL = 0xBB;
+    int TYPE_LITERAL = 0xBC;
+    int EVENT_OBJECT = 0xBD;
+    int COMMENT = 0xBE;
+    int HINT = 0xBF;
+
+    int FIELD_NAME0 = 0xC0;
+    // ...
+    int FIELD_NAME31 = 0xDF;
+
+    int STRING_0 = 0xE0;
+    // ...
+    int STRING_31 = 0xFF;
+
+    String[] STRING_FOR_CODE = _stringForCode(BinaryWireCode.class);
+
+    static String[] _stringForCode(Class clazz) {
+        String[] stringForCode = new String[256];
+        try {
+            for ( Field field : clazz.getDeclaredFields()) {
+                if (field.getType() == int.class)
+                    stringForCode[field.getInt(null)] = field.getName();
+                else if (field.getType() == byte.class)
+                    stringForCode[field.getByte(null) & 0xFF] = field.getName();
+            }
+            for (int i = FIELD_NAME0; i <= FIELD_NAME31; i++)
+                stringForCode[i] = "FIELD_" + i;
+            for (int i = STRING_0; i <= STRING_31; i++)
+                stringForCode[i] = "STRING_" + i;
+            for (int i = 0; i < stringForCode.length; i++) {
+                if (stringForCode[i] == null)
+                    if (i <= ' ' || i >= 127)
+                        stringForCode[i] = "Unknown_0x" + Integer.toHexString(i).toUpperCase();
+                    else
+                        stringForCode[i] = "Unknown_" + (char) i;
+            }
+        } catch (IllegalAccessException e) {
+            throw new AssertionError(e);
+        }
+        return stringForCode;
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/ByteStringAppender.java

@@ -0,0 +1,308 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.Maths;
+import net.openhft.chronicle.core.io.IORuntimeException;
+import net.openhft.chronicle.core.io.UnsafeText;
+import org.jetbrains.annotations.NotNull;
+
+import java.io.Writer;
+import java.math.BigDecimal;
+import java.nio.BufferOverflowException;
+import java.nio.BufferUnderflowException;
+
+/**
+ * Methods to append text to a Bytes. This extends the Appendable interface.
+ */
+@SuppressWarnings({"rawtypes", "unchecked"})
+public interface ByteStringAppender<B extends ByteStringAppender<B>> extends StreamingDataOutput<B>, Appendable {
+
+    /**
+     * @return these Bytes as a Writer
+     */
+
+    default Writer writer() {
+        return new ByteStringWriter(this);
+    }
+
+    /**
+     * Append a char in UTF-8
+     *
+     * @param ch to append
+     * @return this
+     * @throws BufferUnderflowException if the capacity of the underlying buffer was exceeded
+     */
+    @Override
+
+    default B append(char ch) throws BufferOverflowException {
+        BytesInternal.appendUtf8Char(this, ch);
+        return (B) this;
+    }
+
+    /**
+     * Append a characters in UTF-8
+     *
+     * @param cs to append
+     * @return this
+     * @throws BufferUnderflowException if the capacity of the underlying buffer was exceeded
+     */
+    @Override
+
+    default B append( CharSequence cs) throws BufferOverflowException {
+        if (cs.length() == 0)
+            return (B) this;
+        try {
+            return append(cs, 0, cs.length());
+        } catch (IndexOutOfBoundsException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+    /**
+     * Append a boolean as T or F
+     *
+     * @param flag to append
+     * @return this
+     * @throws BufferUnderflowException if the capacity of the underlying buffer was exceeded
+     * @throws IORuntimeException       if an error occurred while attempting to resize the underlying buffer
+     */
+
+    default B append(boolean flag) throws BufferOverflowException {
+        return append(flag ? 'T' : 'F');
+    }
+
+    /**
+     * Append an int in decimal
+     *
+     * @param value to append
+     * @return this
+     * @throws BufferUnderflowException if the capacity of the underlying buffer was exceeded
+     * @throws IORuntimeException       if an error occurred while attempting to resize the underlying buffer
+     */
+
+    default B append(int value) throws BufferOverflowException {
+        BytesInternal.appendBase10(this, value);
+        return (B) this;
+    }
+
+    /**
+     * Append a long in decimal
+     *
+     * @param value to append
+     * @return this
+     * @throws BufferUnderflowException if the capacity of the underlying buffer was exceeded
+     * @throws IORuntimeException       if an error occurred while attempting to resize the underlying buffer
+     */
+
+    default B append(long value) throws BufferOverflowException {
+        if (value == (int) value)
+            BytesInternal.appendBase10(this, (int) value);
+        else
+            BytesInternal.appendBase10(this, value);
+        return (B) this;
+    }
+
+
+    default B appendBase(long value, int base) throws BufferOverflowException {
+        BytesInternal.append(this, value, base);
+        return (B) this;
+    }
+
+
+    default B appendBase16(long value) throws BufferOverflowException {
+        BytesInternal.appendBase16(this, value, 1);
+        return (B) this;
+    }
+
+
+    default B appendBase16(long value, int minDigits) throws BufferOverflowException {
+        BytesInternal.appendBase16(this, value, minDigits);
+        return (B) this;
+    }
+
+    /**
+     * Append a long in decimal with a given number of decimal places. Print value * 10^-decimalPlaces
+     *
+     * @param value         to append
+     * @param decimalPlaces to shift the decimal place.
+     * @return this
+     * @throws BufferUnderflowException if the capacity of the underlying buffer was exceeded
+     * @throws IORuntimeException       if an error occurred while attempting to resize the underlying buffer
+     */
+
+    default B appendDecimal(long value, int decimalPlaces) throws BufferOverflowException {
+        BytesInternal.appendDecimal(this, value, decimalPlaces);
+        return (B) this;
+    }
+
+    /**
+     * Append a float in decimal notation
+     *
+     * @param f to append
+     * @return this
+     * @throws BufferUnderflowException if the capacity of the underlying buffer was exceeded
+     * @throws IORuntimeException       if an error occurred while attempting to resize the underlying buffer
+     */
+
+    default B append(float f) throws BufferOverflowException {
+        float f2 = Math.abs(f);
+        if (f2 > 1e6 || f2 < 1e-3) {
+            return append(Float.toString(f));
+        }
+        int precision = (int) Math.floor(6 - Math.log10(f2));
+        long tens = Maths.tens(precision);
+        return append((double) Math.round(f * tens) / tens);
+    }
+
+    /**
+     * Append a double in decimal notation
+     *
+     * @param d to append
+     * @return this
+     * @throws BufferUnderflowException if the capacity of the underlying buffer was exceeded
+     * @throws IORuntimeException       if an error occurred while attempting to resize the underlying buffer
+     */
+
+    default B append(double d) throws BufferOverflowException {
+        BytesInternal.append(this, d);
+        return (B) this;
+    }
+
+    /**
+     * Append a double in decimal notation to a specific number of decimal places. Trailing zeros are not truncated.
+     * <p>
+     * If the number would normally be printed with more decimal places, the number is rounded.
+     *
+     * @param d             to append
+     * @param decimalPlaces to always produce
+     * @return this
+     * @throws BufferUnderflowException if the capacity of the underlying buffer was exceeded
+     * @throws IORuntimeException       if an error occurred while attempting to resize the underlying buffer
+     */
+
+    default B append(double d, int decimalPlaces) throws BufferOverflowException {
+        if (decimalPlaces < 0)
+            throw new IllegalArgumentException();
+        if (decimalPlaces < 18) {
+            double d2 = d * Maths.tens(decimalPlaces);
+            if (d2 < Long.MAX_VALUE && d2 > Long.MIN_VALUE) {
+                // changed from java.lang.Math.round(d2) as this was shown up to cause latency
+                long round = d2 > 0.0 ? (long) (d2 + 0.5) : (long) (d2 - 0.5);
+                if (canWriteDirect(20 + decimalPlaces)) {
+                    long address = addressForWritePosition();
+                    long address2 = UnsafeText.appendBase10d(address, round, decimalPlaces);
+                    writeSkip(address2 - address);
+                } else {
+                    appendDecimal(round, decimalPlaces);
+                }
+                return (B) this;
+            }
+        }
+        return append(d);
+    }
+
+    /**
+     * Append a portion of a String to the Bytes in UTF-8.
+     *
+     * @param cs    to copy
+     * @param start index of the first char inclusive
+     * @param end   index of the last char exclusive.
+     * @return this
+     * @throws BufferUnderflowException if the capacity of the underlying buffer was exceeded
+     */
+    @Override
+
+    default B append( CharSequence cs, int start, int end)
+            throws IndexOutOfBoundsException, BufferOverflowException {
+        BytesInternal.appendUtf8(this, cs, start, end - start);
+        return (B) this;
+    }
+
+    /**
+     * Append a String to the Bytes in ISO-8859-1
+     *
+     * @param cs to write
+     * @return this
+     * @throws BufferOverflowException  If the string as too large to write in the capacity available
+     * @throws BufferUnderflowException if the capacity of the underlying buffer was exceeded
+     */
+
+    default B append8bit( CharSequence cs)
+            throws BufferOverflowException, BufferUnderflowException, IndexOutOfBoundsException {
+        return append8bit(cs, 0, cs.length());
+    }
+
+    default B append8bit( BytesStore bs)
+            throws BufferOverflowException, BufferUnderflowException, IndexOutOfBoundsException {
+        return write(bs, 0L, bs.readRemaining());
+    }
+
+    default B append8bit( String cs)
+            throws BufferOverflowException, BufferUnderflowException, IndexOutOfBoundsException {
+        return append8bit(cs, 0, cs.length());
+    }
+
+    /**
+     * Append a portion of a String to the Bytes in ISO-8859-1
+     *
+     * @param cs    to copy
+     * @param start index of the first char inclusive
+     * @param end   index of the last char exclusive.
+     * @return this
+     * @throws BufferOverflowException   If the string as too large to write in the capacity available
+     * @throws BufferUnderflowException  if the capacity of the underlying buffer was exceeded
+     * @throws IndexOutOfBoundsException if the start or the end are not valid for the CharSequence
+     */
+    default B append8bit( CharSequence cs, int start, int end)
+            throws IllegalArgumentException, BufferOverflowException, BufferUnderflowException, IndexOutOfBoundsException {
+        if (cs instanceof BytesStore) {
+            return write((BytesStore) cs, (long) start, end);
+        }
+        for (int i = start; i < end; i++) {
+            char c = cs.charAt(i);
+            if (c > 255) c = '?';
+            writeByte((byte) c);
+        }
+        return (B) this;
+    }
+
+    default B append8bit( BytesStore bs, long start, long end)
+            throws IllegalArgumentException, BufferOverflowException, BufferUnderflowException, IndexOutOfBoundsException {
+        return write(bs, start, end);
+    }
+
+
+    default B appendDateMillis(long dateInMillis) {
+        BytesInternal.appendDateMillis(this, dateInMillis);
+        return (B) this;
+    }
+
+
+    default B appendTimeMillis(long timeOfDayInMillis) {
+        BytesInternal.appendTimeMillis(this, timeOfDayInMillis % 86400_000L);
+        return (B) this;
+    }
+
+
+    default B append( BigDecimal bigDecimal) {
+        append(bigDecimal.toString());
+        return (B) this;
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/ByteStringParser.java

@@ -0,0 +1,252 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.Maths;
+import net.openhft.chronicle.core.annotation.ForceInline;
+import net.openhft.chronicle.core.io.IORuntimeException;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+import java.io.Reader;
+import java.math.BigDecimal;
+import java.nio.BufferUnderflowException;
+
+/**
+ * Supports parsing bytes as text.  You can parse them as special or white space terminated text.
+ */
+interface ByteStringParser<B extends ByteStringParser<B>> extends StreamingDataInput<B> {
+    /**
+     * Access these bytes as an ISO-8859-1 encoded Reader
+     *
+     * @return as a Reader
+     */
+
+    default Reader reader() {
+        return new ByteStringReader(this);
+    }
+
+    /**
+     * Return true or false, or null if it could not be detected
+     * as true or false.  Case is not important
+     * <p>
+     * <p>false: f, false, n, no, 0
+     * <p>
+     * <p>true: t, true, y, yes, 1
+     *
+     * @param tester to detect the end of the text.
+     * @return true, false, or null if neither.
+     */
+
+    default Boolean parseBoolean( StopCharTester tester) {
+        return BytesInternal.parseBoolean(this, tester);
+    }
+
+
+    default Boolean parseBoolean() {
+        return BytesInternal.parseBoolean(this, StopCharTesters.NON_ALPHA_DIGIT);
+    }
+
+    /**
+     * parse text with UTF-8 decoding as character terminated.
+     *
+     * @param stopCharTester to check if the end has been reached.
+     * @return the text as a String.
+     */
+
+    @ForceInline
+    default String parseUtf8( StopCharTester stopCharTester) {
+        return BytesInternal.parseUtf8(this, stopCharTester);
+    }
+
+
+    @Deprecated(/* to be removed in x.22 */)
+    default String parseUTF( StopCharTester stopCharTester) {
+        return parseUtf8(stopCharTester);
+    }
+
+    /**
+     * parse text with UTF-8 decoding as character terminated.
+     *
+     * @param buffer         to populate
+     * @param stopCharTester to check if the end has been reached.
+     */
+    @ForceInline
+    default void parseUtf8( Appendable buffer,  StopCharTester stopCharTester) throws BufferUnderflowException {
+        BytesInternal.parseUtf8(this, buffer, stopCharTester);
+    }
+
+    @Deprecated(/* to be removed in x.22 */)
+    default void parseUTF( Appendable buffer,  StopCharTester stopCharTester) throws BufferUnderflowException {
+        parseUtf8(buffer, stopCharTester);
+    }
+
+    /**
+     * parse text with UTF-8 decoding as one or two character terminated.
+     *
+     * @param buffer          to populate
+     * @param stopCharsTester to check if the end has been reached.
+     */
+    @ForceInline
+    default void parseUtf8( Appendable buffer,  StopCharsTester stopCharsTester)
+            throws BufferUnderflowException, IORuntimeException {
+        BytesInternal.parseUtf8(this, buffer, stopCharsTester);
+    }
+
+    @Deprecated(/* to be removed in x.22 */)
+    default void parseUTF( Appendable buffer,  StopCharsTester stopCharsTester)
+            throws BufferUnderflowException, IORuntimeException {
+        parseUtf8(buffer, stopCharsTester);
+    }
+
+    /**
+     * parse text with ISO-8859-1 decoding as character terminated.
+     *
+     * @param buffer         to populate
+     * @param stopCharTester to check if the end has been reached.
+     */
+    @SuppressWarnings("rawtypes")
+    @ForceInline
+    default void parse8bit(Appendable buffer,  StopCharTester stopCharTester)
+            throws BufferUnderflowException {
+        if (buffer instanceof StringBuilder)
+            BytesInternal.parse8bit(this, (StringBuilder) buffer, stopCharTester);
+        else
+            BytesInternal.parse8bit(this, (Bytes) buffer, stopCharTester);
+    }
+
+    /**
+     * parse text with ISO-8859-1 decoding as character terminated.
+     *
+     * @param stopCharTester to check if the end has been reached.
+     */
+    default String parse8bit( StopCharTester stopCharTester)
+            throws BufferUnderflowException {
+        return BytesInternal.parse8bit(this, stopCharTester);
+    }
+
+    /**
+     * parse text with ISO-8859-1 decoding as character terminated.
+     *
+     * @param buffer          to populate
+     * @param stopCharsTester to check if the end has been reached.
+     */
+    @SuppressWarnings("rawtypes")
+    @ForceInline
+    default void parse8bit(Appendable buffer,  StopCharsTester stopCharsTester)
+            throws BufferUnderflowException {
+        if (buffer instanceof StringBuilder)
+            BytesInternal.parse8bit(this, (StringBuilder) buffer, stopCharsTester);
+        else
+            BytesInternal.parse8bit(this, (Bytes) buffer, stopCharsTester);
+    }
+
+    @SuppressWarnings("rawtypes")
+    default void parse8bit(Bytes buffer,  StopCharsTester stopCharsTester)
+            throws BufferUnderflowException {
+        BytesInternal.parse8bit(this, buffer, stopCharsTester);
+    }
+
+    default void parse8bit(StringBuilder buffer,  StopCharsTester stopCharsTester)
+            throws BufferUnderflowException {
+        BytesInternal.parse8bit(this, buffer, stopCharsTester);
+    }
+
+    /**
+     * parse text as an int. The terminating character is consumed.
+     *
+     * @return an int.
+     */
+    @ForceInline
+    default int parseInt() throws BufferUnderflowException {
+        return Maths.toInt32(BytesInternal.parseLong(this));
+    }
+
+    /**
+     * parse text as a long integer. The terminating character is consumed.
+     *
+     * @return a long.
+     */
+    @ForceInline
+    default long parseLong() throws BufferUnderflowException {
+        return BytesInternal.parseLong(this);
+    }
+
+    /**
+     * parse text as a float decimal. The terminating character is consumed.
+     * <p>
+     * The number of decimal places can be retrieved with  lastDecimalPlaces()
+     *
+     * @return a float.
+     */
+    default float parseFloat() throws BufferUnderflowException {
+        return (float) BytesInternal.parseDouble(this);
+    }
+
+    /**
+     * parse text as a double decimal. The terminating character is consumed.
+     * <p>
+     * The number of decimal places can be retrieved with  lastDecimalPlaces()
+     *
+     * @return a double.
+     */
+    default double parseDouble() throws BufferUnderflowException {
+        return BytesInternal.parseDouble(this);
+    }
+
+    /**
+     * Parse the significant digits of a decimal number.
+     * <p>
+     * The number of decimal places can be retrieved with  lastDecimalPlaces()
+     *
+     * @return the significant digits
+     */
+    default long parseLongDecimal() throws BufferUnderflowException {
+        return BytesInternal.parseLongDecimal(this);
+    }
+
+    /**
+     * @return the last number of decimal places for parseDouble or parseLongDecimal
+     */
+    int lastDecimalPlaces();
+
+    /**
+     * Store the last number of decimal places. If
+     *
+     * @param lastDecimalPlaces set the number of decimal places if positive, otherwise 0.
+     */
+    void lastDecimalPlaces(int lastDecimalPlaces);
+
+    /**
+     * Skip text until a terminating character is reached.
+     *
+     * @param tester to stop at
+     * @return true if a terminating character was found, false if the end of the buffer was reached.
+     */
+    @ForceInline
+    default boolean skipTo( StopCharTester tester) {
+        return BytesInternal.skipTo(this, tester);
+    }
+
+
+    default BigDecimal parseBigDecimal() {
+        return new BigDecimal(parseUtf8(StopCharTesters.NUMBER_END));
+    }
+
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/ByteStringReader.java

@@ -0,0 +1,66 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import java.io.IOException;
+import java.io.Reader;
+import java.nio.BufferUnderflowException;
+
+/**
+ * A Reader wrapper for Bytes.  This Reader moves the readPosition() of the underlying Bytes up to the readLimit()
+ */
+@SuppressWarnings("rawtypes")
+public class ByteStringReader extends Reader {
+    private final ByteStringParser in;
+
+    public ByteStringReader(ByteStringParser in) {
+        this.in = in;
+    }
+
+    @Override
+    public int read() {
+        return in.readRemaining() > 0 ? in.readUnsignedByte() : -1;
+    }
+
+    @Override
+    public long skip(long n) throws IOException {
+        long len = Math.min(in.readRemaining(), n);
+        try {
+            in.readSkip(len);
+
+        } catch (BufferUnderflowException e) {
+            throw new IOException(e);
+        }
+        return len;
+    }
+
+    @Override
+    public int read(char[] cbuf, int off, int len) throws IOException {
+        try {
+            return in.read(cbuf, off, len);
+
+        } catch (BufferUnderflowException e) {
+            throw new IOException(e);
+        }
+    }
+
+    @Override
+    public void close() {
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/ByteStringWriter.java

@@ -0,0 +1,94 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import org.jetbrains.annotations.NotNull;
+
+import java.io.IOException;
+import java.io.Writer;
+import java.nio.BufferOverflowException;
+
+/**
+ * A Writer for an underlying Bytes.  This moves the writePosition() up to the writeLimit();
+ */
+@SuppressWarnings("rawtypes")
+class ByteStringWriter extends Writer {
+    private final ByteStringAppender out;
+
+    ByteStringWriter(ByteStringAppender out) {
+        this.out = out;
+    }
+
+    @Override
+    public void write(int c) throws IOException {
+        try {
+            out.append((char) c);
+
+        } catch ( BufferOverflowException e) {
+            throw new IOException(e);
+        }
+    }
+
+    @Override
+    public void write( String str) throws IOException {
+        out.append(str);
+    }
+
+    @Override
+    public void write( String str, int off, int len) throws IOException {
+        out.append(str, off, off + len);
+    }
+
+
+    @Override
+    public Writer append( CharSequence csq) throws IOException {
+        out.append(csq);
+        return this;
+    }
+
+
+    @Override
+    public Writer append( CharSequence csq, int start, int end) throws IOException {
+        out.append(csq, start, end);
+        return this;
+    }
+
+
+    @Override
+    public Writer append(char c) throws IOException {
+        out.append(c);
+        return this;
+    }
+
+    @Override
+    public void flush() {
+
+    }
+
+    @Override
+    public void close() {
+
+    }
+
+    @Override
+    public void write(char[] cbuf, int off, int len) throws IOException {
+        for (int i = 0; i < len; i++)
+            out.append(cbuf[i + off]);
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/Byteable.java

@@ -0,0 +1,52 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import org.jetbrains.annotations.Nullable;
+
+import java.nio.BufferOverflowException;
+import java.nio.BufferUnderflowException;
+
+/**
+ * This Interface allows a reference to off heap memory to be reassigned.
+ * <p></p>
+ * A reference to off heap memory is a proxy for some memory which sits outside the heap.
+ */
+public interface Byteable<B extends BytesStore<B, Underlying>, Underlying> {
+    /**
+     * This setter for a data type which points to an underlying ByteStore.
+     *
+     * @param bytesStore the fix point ByteStore
+     * @param offset     the offset within the ByteStore
+     * @param length     the length in the ByteStore
+     */
+    void bytesStore(BytesStore<B, Underlying> bytesStore, long offset, long length)
+            throws IllegalStateException, IllegalArgumentException, BufferOverflowException,
+            BufferUnderflowException;
+
+
+    BytesStore<B, Underlying> bytesStore();
+
+    long offset();
+
+    /**
+     * @return the maximum size in byte for this reference.
+     */
+    long maxSize();
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/Bytes.java

@@ -0,0 +1,944 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.Maths;
+import net.openhft.chronicle.core.annotation.UsedViaReflection;
+import net.openhft.chronicle.core.io.IORuntimeException;
+import net.openhft.chronicle.core.io.ReferenceOwner;
+import net.openhft.chronicle.core.util.ObjectUtils;
+import net.openhft.chronicle.core.util.StringUtils;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.BufferOverflowException;
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+
+/**
+ * Bytes is a pointer to a region of memory within a BytesStore. It can be for a fixed region of
+ * memory or an "elastic" buffer which can be resized, but not for a fixed region. <p></p> This is a
+ * BytesStore which is mutable and not thread safe. It has a write position and read position which
+ * must follow these constraints <p></p> start() &lt;= readPosition() &lt;= writePosition() &lt;=
+ * writeLimit() &lt;= capacity() <p></p> Also readLimit() == writePosition() and readPosition()
+ * &lt;= safeLimit(); <p></p>
+ */
+@SuppressWarnings({"rawtypes", "unchecked"})
+public interface Bytes<Underlying> extends
+        BytesStore<Bytes<Underlying>, Underlying>,
+        BytesIn<Underlying>,
+        BytesOut<Underlying> {
+
+    long MAX_CAPACITY = Long.MAX_VALUE & ~0xF; // 8 EiB - 16
+    int MAX_HEAP_CAPACITY = Integer.MAX_VALUE & ~0xF;  // 2 GiB - 16
+    @Deprecated(/* to be removed in x.22 */)
+    int MAX_BYTE_BUFFER_CAPACITY = MAX_HEAP_CAPACITY;
+    int DEFAULT_BYTE_BUFFER_CAPACITY = 256;
+
+    /**
+     * Creates and returns a new elastic wrapper for a direct (off-heap) ByteBuffer with a default capacity
+     * which will be resized as required.
+     *
+     * @return a new elastic wrapper for a direct (off-heap) ByteBuffer with a default capacity
+     * which will be resized as required
+     */
+
+    static Bytes<ByteBuffer> elasticByteBuffer() {
+        return elasticByteBuffer(DEFAULT_BYTE_BUFFER_CAPACITY);
+    }
+
+    /**
+     * Creates and returns a new elastic wrapper for a direct (off-heap) ByteBuffer with
+     * the given {@code initialCapacity} which will be resized as required.
+     *
+     * @param initialCapacity the initial non-negative capacity given in bytes
+     * @return a new elastic wrapper for a direct (off-heap) ByteBuffer with
+     * the given {@code initialCapacity} which will be resized as required
+     */
+
+    static Bytes<ByteBuffer> elasticByteBuffer(int initialCapacity) {
+        return elasticByteBuffer(initialCapacity, MAX_HEAP_CAPACITY);
+    }
+
+    /**
+     * Creates and returns a new elastic wrapper for a direct (off-heap) ByteBuffer with
+     * the given {@code initialCapacity} which will be resized as required up
+     * to the given {@code maxSize}.
+     *
+     * @param initialCapacity the initial non-negative capacity given in bytes
+     * @param maxCapacity     the max capacity given in bytes equal or greater than initialCapacity
+     * @return a new elastic wrapper for a direct (off-heap) ByteBuffer with
+     * the given {@code initialCapacity} which will be resized as required up
+     * to the given {@code maxCapacity}
+     */
+
+    static Bytes<ByteBuffer> elasticByteBuffer(int initialCapacity, int maxCapacity) {
+         NativeBytesStore<ByteBuffer> bs = NativeBytesStore.elasticByteBuffer(initialCapacity, maxCapacity);
+        try {
+            return bs.bytesForWrite();
+        } finally {
+            bs.release(ReferenceOwner.INIT);
+        }
+    }
+
+    /**
+     * Creates and returns a new elastic wrapper for a heap ByteBuffer with
+     * the given {@code initialCapacity} which will be resized as required.
+     *
+     * @param initialCapacity the initial non-negative capacity given in bytes
+     * @return a new elastic wrapper for a heap ByteBuffer with
+     * the given {@code initialCapacity} which will be resized as required
+     */
+
+    static Bytes<ByteBuffer> elasticHeapByteBuffer(int initialCapacity) {
+         HeapBytesStore<ByteBuffer> bs = HeapBytesStore.wrap(ByteBuffer.allocate(initialCapacity));
+        try {
+            return NativeBytes.wrapWithNativeBytes(bs, Bytes.MAX_HEAP_CAPACITY);
+        } finally {
+            bs.release(INIT);
+        }
+    }
+
+
+    static Bytes<ByteBuffer> elasticHeapByteBuffer() {
+        return elasticHeapByteBuffer(128);
+    }
+
+    /**
+     * Wrap the ByteBuffer ready for reading
+     * Method for convenience only - might not be ideal for performance (creates garbage).
+     * To avoid garbage, use something like this example:
+     * <pre>{@code
+     * import net.openhft.chronicle.bytes.Bytes;
+     * import java.nio.ByteBuffer;
+     *
+     * public class ChronicleBytesWithByteBufferExampleTest {
+     *     private static final String HELLO_WORLD = "hello world";
+     *
+     *     public static void main(String[] args) throws InterruptedException {
+     *         //setup Bytes and ByteBuffer to write from
+     *         Bytes b = Bytes.elasticByteBuffer();
+     *         ByteBuffer toWriteFrom = ByteBuffer.allocate(HELLO_WORLD.length());
+     *         toWriteFrom.put(HELLO_WORLD.getBytes(), 0, HELLO_WORLD.length());
+     *         toWriteFrom.flip();
+     *         byte[] toReadTo = new byte[HELLO_WORLD.length()];
+     *
+     *         doWrite(b, toWriteFrom);
+     *         ByteBuffer byteBuffer = doRead(b);
+     *
+     *         //check result
+     *         final StringBuilder sb = new StringBuilder();
+     *         for (int i = 0; i < HELLO_WORLD.length(); i++) {
+     *             sb.append((char) byteBuffer.get());
+     *         }
+     *         assert sb.toString().equals(HELLO_WORLD): "Failed - strings not equal!";
+     *     }
+     *
+     *     private static void doWrite(Bytes b, ByteBuffer toWrite) {
+     *         //no garbage when writing to Bytes from ByteBuffer
+     *         b.clear();
+     *         b.write(b.writePosition(), toWrite, toWrite.position(), toWrite.limit());
+     *     }
+     *
+     *     private static ByteBuffer doRead(Bytes b) {
+     *         //no garbage when getting the underlying ByteBuffer
+     *         assert b.underlyingObject() instanceof ByteBuffer;
+     *         ByteBuffer byteBuffer = (ByteBuffer) b.underlyingObject();
+     *         return byteBuffer;
+     *     }
+     * }
+     * }</pre>
+     *
+     * @param byteBuffer to wrap
+     * @return a Bytes which wraps the provided ByteBuffer and is ready for reading.
+     */
+
+    static Bytes<ByteBuffer> wrapForRead( ByteBuffer byteBuffer) {
+        BytesStore<?, ByteBuffer> bs = BytesStore.wrap(byteBuffer);
+        try {
+            Bytes<ByteBuffer> bbb = bs.bytesForRead();
+            bbb.readLimit(byteBuffer.limit());
+            bbb.readPosition(byteBuffer.position());
+            return bbb;
+        } finally {
+            bs.release(INIT);
+        }
+    }
+
+    /**
+     * Wrap the ByteBuffer ready for writing
+     * Method for convenience only - might not be ideal for performance (creates garbage).
+     * To avoid garbage, use something like this example:
+     * <pre>{@code
+     * import net.openhft.chronicle.bytes.Bytes;
+     * import java.nio.ByteBuffer;
+     *
+     * public class ChronicleBytesWithByteBufferExampleTest {
+     *     private static final String HELLO_WORLD = "hello world";
+     *
+     *     public static void main(String[] args) throws InterruptedException {
+     *         //setup Bytes and ByteBuffer to write from
+     *         Bytes b = Bytes.elasticByteBuffer();
+     *         ByteBuffer toWriteFrom = ByteBuffer.allocate(HELLO_WORLD.length());
+     *         toWriteFrom.put(HELLO_WORLD.getBytes(), 0, HELLO_WORLD.length());
+     *         toWriteFrom.flip();
+     *         byte[] toReadTo = new byte[HELLO_WORLD.length()];
+     *
+     *         doWrite(b, toWriteFrom);
+     *         ByteBuffer byteBuffer = doRead(b);
+     *
+     *         //check result
+     *         final StringBuilder sb = new StringBuilder();
+     *         for (int i = 0; i < HELLO_WORLD.length(); i++) {
+     *             sb.append((char) byteBuffer.get());
+     *         }
+     *         assert sb.toString().equals(HELLO_WORLD): "Failed - strings not equal!";
+     *     }
+     *
+     *     private static void doWrite(Bytes b, ByteBuffer toWrite) {
+     *         //no garbage when writing to Bytes from ByteBuffer
+     *         b.clear();
+     *         b.write(b.writePosition(), toWrite, toWrite.position(), toWrite.limit());
+     *     }
+     *
+     *     private static ByteBuffer doRead(Bytes b) {
+     *         //no garbage when getting the underlying ByteBuffer
+     *         assert b.underlyingObject() instanceof ByteBuffer;
+     *         ByteBuffer byteBuffer = (ByteBuffer) b.underlyingObject();
+     *         return byteBuffer;
+     *     }
+     * }
+     * }</pre>
+     *
+     * @param byteBuffer to wrap
+     * @return a Bytes which wraps the provided ByteBuffer and is ready for writing.
+     */
+
+    static Bytes<ByteBuffer> wrapForWrite( ByteBuffer byteBuffer) {
+        BytesStore<?, ByteBuffer> bs = BytesStore.wrap(byteBuffer);
+        try {
+            Bytes<ByteBuffer> bbb = bs.bytesForWrite();
+            bbb.writePosition(byteBuffer.position());
+            bbb.writeLimit(byteBuffer.limit());
+            return bbb;
+        } finally {
+            bs.release(INIT);
+        }
+    }
+
+    /**
+     * Wrap the byte[] ready for reading
+     * Method for convenience only - might not be ideal for performance (creates garbage).
+     * To avoid garbage, use something like this example:
+     * <pre>{@code
+     * import net.openhft.chronicle.bytes.Bytes;
+     * import java.nio.charset.Charset;
+     *
+     * public class ChronicleBytesWithPrimByteArrayExampleTest {
+     *     private static final Charset ISO_8859 = Charset.forName("ISO-8859-1");
+     *     private static final String HELLO_WORLD = "hello world";
+     *
+     *     public static void main(String[] args) {
+     *         //setup Bytes and byte[]s to write from and read to
+     *         Bytes b = Bytes.elasticByteBuffer();
+     *         byte[] toWriteFrom = HELLO_WORLD.getBytes(ISO_8859);
+     *         byte[] toReadTo = new byte[HELLO_WORLD.length()];
+     *
+     *         doWrite(b, toWriteFrom);
+     *         doRead(b, toReadTo);
+     *
+     *         //check result
+     *         final StringBuilder sb = new StringBuilder();
+     *         for (int i = 0; i < HELLO_WORLD.length(); i++) {
+     *             sb.append((char) toReadTo[i]);
+     *         }
+     *         assert sb.toString().equals(HELLO_WORLD): "Failed - strings not equal!";
+     *     }
+     *
+     *     private static void doWrite(Bytes b, byte[] toWrite) {
+     *         //no garbage when writing to Bytes from byte[]
+     *         b.clear();
+     *         b.write(toWrite);
+     *     }
+     *
+     *     private static void doRead(Bytes b, byte[] toReadTo) {
+     *         //no garbage when reading from Bytes into byte[]
+     *         b.read( toReadTo, 0, HELLO_WORLD.length());
+     *     }
+     * }
+     * }</pre>
+     *
+     * @param byteArray to wrap
+     * @return the Bytes ready for reading.
+     */
+
+    static Bytes<byte[]> wrapForRead( byte[] byteArray) {
+         HeapBytesStore<byte[]> bs = BytesStore.wrap(byteArray);
+        try {
+            return bs.bytesForRead();
+        } finally {
+            bs.release(INIT);
+        }
+    }
+
+    /**
+     * Wrap the byte[] ready for writing
+     * Method for convenience only - might not be ideal for performance (creates garbage).
+     * To avoid garbage, use something like this example:
+     * <pre>{@code
+     * import net.openhft.chronicle.bytes.Bytes;
+     * import java.nio.charset.Charset;
+     *
+     * public class ChronicleBytesWithPrimByteArrayExampleTest {
+     *     private static final Charset ISO_8859 = Charset.forName("ISO-8859-1");
+     *     private static final String HELLO_WORLD = "hello world";
+     *
+     *     public static void main(String[] args) {
+     *         //setup Bytes and byte[]s to write from and read to
+     *         Bytes b = Bytes.elasticByteBuffer();
+     *         byte[] toWriteFrom = HELLO_WORLD.getBytes(ISO_8859);
+     *         byte[] toReadTo = new byte[HELLO_WORLD.length()];
+     *
+     *         doWrite(b, toWriteFrom);
+     *         doRead(b, toReadTo);
+     *
+     *         //check result
+     *         final StringBuilder sb = new StringBuilder();
+     *         for (int i = 0; i < HELLO_WORLD.length(); i++) {
+     *             sb.append((char) toReadTo[i]);
+     *         }
+     *         assert sb.toString().equals(HELLO_WORLD): "Failed - strings not equal!";
+     *     }
+     *
+     *     private static void doWrite(Bytes b, byte[] toWrite) {
+     *         //no garbage when writing to Bytes from byte[]
+     *         b.clear();
+     *         b.write(toWrite);
+     *     }
+     *
+     *     private static void doRead(Bytes b, byte[] toReadTo) {
+     *         //no garbage when reading from Bytes into byte[]
+     *         b.read( toReadTo, 0, HELLO_WORLD.length());
+     *     }
+     * }
+     * }</pre>
+     *
+     * @param byteArray to wrap
+     * @return the Bytes ready for writing.
+     */
+
+    static Bytes<byte[]> wrapForWrite( byte[] byteArray) {
+         BytesStore bs = BytesStore.wrap(byteArray);
+        try {
+            return bs.bytesForWrite();
+        } finally {
+            bs.release(INIT);
+        }
+    }
+
+    /**
+     * Convert text to bytes using ISO-8859-1 encoding and return a Bytes ready for reading.
+     * <p>
+     * Note: this returns a direct Bytes now
+     *
+     * @param text to convert
+     * @return Bytes ready for reading.
+     */
+
+    static Bytes<?> from( CharSequence text) {
+        return from(text.toString());
+    }
+
+    @Deprecated(/* to be removed in x.22 */)
+    static Bytes<?> fromString(String text) {
+        return from(text);
+    }
+
+    /**
+     * Convert text to bytes using ISO-8859-1 encoding and return a Bytes ready for reading.
+     * <p>
+     * Note: this returns a direct Bytes now
+     *
+     * @param text to convert
+     * @return Bytes ready for reading.
+     */
+
+    static Bytes<?> directFrom( String text) {
+        NativeBytesStore from = NativeBytesStore.from(text);
+        try {
+            return from.bytesForRead();
+        } finally {
+            from.release(INIT);
+        }
+    }
+
+    /**
+     * Convert text to bytes using ISO-8859-1 encoding and return a Bytes ready for reading.
+     * <p>
+     * Note: this returns a heap Bytes
+     *
+     * @param text to convert
+     * @return Bytes ready for reading.
+     */
+
+    static Bytes<byte[]> from( String text) throws IllegalArgumentException, IllegalStateException {
+        return wrapForRead(text.getBytes(StandardCharsets.ISO_8859_1));
+    }
+
+    @UsedViaReflection
+    static Bytes<byte[]> valueOf(String text) {
+        return from(text);
+    }
+
+    /**
+     * Creates and returns a new fix sized wrapper for native (64-bit address)
+     * memory with the given {@code capacity}.
+     *
+     * @param capacity the non-negative capacity given in bytes
+     * @return a new fix sized wrapper for native (64-bit address)
+     * memory with the given {@code capacity}
+     */
+
+    static VanillaBytes<Void> allocateDirect(long capacity) throws IllegalArgumentException {
+         NativeBytesStore<Void> bs = NativeBytesStore.nativeStoreWithFixedCapacity(capacity);
+        try {
+            return new VanillaBytes<>(bs);
+        } finally {
+            bs.release(INIT);
+        }
+    }
+
+    /**
+     * Creates and returns a new elastic wrapper for native (64-bit address)
+     * memory with zero initial capacity which will be resized as required.
+     *
+     * @return a new elastic wrapper for native (64-bit address)
+     * memory with zero initial capacity which will be resized as required
+     */
+
+    static NativeBytes<Void> allocateElasticDirect() {
+        return NativeBytes.nativeBytes();
+    }
+
+    /**
+     * Creates and returns a new elastic wrapper for native (64-bit address)
+     * memory with the given {@code initialCapacity} which will be resized as required.
+     *
+     * @param initialCapacity the initial non-negative capacity given in bytes
+     * @return a new elastic wrapper for native (64-bit address)
+     * memory with the given {@code initialCapacity} which will be resized as required
+     */
+
+    static NativeBytes<Void> allocateElasticDirect(long initialCapacity) throws IllegalArgumentException {
+        return NativeBytes.nativeBytes(initialCapacity);
+    }
+
+
+    static OnHeapBytes allocateElasticOnHeap() {
+        return allocateElasticOnHeap(32);
+    }
+
+
+    static OnHeapBytes allocateElasticOnHeap(int initialCapacity) {
+        HeapBytesStore<byte[]> wrap = HeapBytesStore.wrap(new byte[initialCapacity]);
+        try {
+            return new OnHeapBytes(wrap, true);
+        } finally {
+            wrap.release(INIT);
+        }
+    }
+
+    /**
+     * Creates a string from the {@code position} to the {@code limit}, The buffer is not modified
+     * by this call
+     *
+     * @param buffer the buffer to use
+     * @return a string contain the text from the {@code position}  to the  {@code limit}
+     */
+
+    static String toString( final Bytes<?> buffer) throws BufferUnderflowException {
+        return toString(buffer, MAX_HEAP_CAPACITY);
+    }
+
+    /**
+     * Creates a string from the {@code position} to the {@code limit}, The buffer is not modified
+     * by this call
+     *
+     * @param buffer the buffer to use
+     * @param maxLen of the result returned
+     * @return a string contain the text from the {@code position}  to the  {@code limit}
+     */
+
+    static String toString( final Bytes<?> buffer, long maxLen) throws
+            BufferUnderflowException {
+
+        if (buffer.refCount() < 1)
+            // added because something is crashing the JVM
+            return "<unknown>";
+
+        ReferenceOwner toString = ReferenceOwner.temporary("toString");
+        buffer.reserve(toString);
+        try {
+
+            if (buffer.readRemaining() == 0)
+                return "";
+
+            final long length = Math.min(maxLen + 1, buffer.readRemaining());
+
+             final StringBuilder builder = new StringBuilder();
+            try {
+                buffer.readWithLength(length, b -> {
+                    while (buffer.readRemaining() > 0) {
+                        if (builder.length() >= maxLen) {
+                            builder.append("...");
+                            break;
+                        }
+                        builder.append((char) buffer.readByte());
+                    }
+                });
+            } catch (Exception e) {
+                builder.append(' ').append(e);
+            }
+            return builder.toString();
+        } finally {
+            buffer.release(toString);
+        }
+    }
+
+    /**
+     * The buffer is not modified by this call
+     *
+     * @param buffer   the buffer to use
+     * @param position the position to create the string from
+     * @param len      the number of characters to show in the string
+     * @return a string contain the text from offset {@code position}
+     */
+
+    static String toString( final Bytes buffer, long position, long len)
+            throws BufferUnderflowException {
+        final long pos = buffer.readPosition();
+        final long limit = buffer.readLimit();
+        buffer.readPositionRemaining(position, len);
+
+        try {
+
+             final StringBuilder builder = new StringBuilder();
+            while (buffer.readRemaining() > 0) {
+                builder.append((char) buffer.readByte());
+            }
+
+            // remove the last comma
+            return builder.toString();
+        } finally {
+            buffer.readLimit(limit);
+            buffer.readPosition(pos);
+        }
+    }
+
+    /**
+     * Creates and returns a new fix sized wrapper for native (64-bit address)
+     * memory with the contents copied from the given {@code bytes} array.
+     * <p>
+     * Changes in the given {@code bytes} will not be affected by writes in
+     * the returned wrapper or vice versa.
+     *
+     * @param bytes array to copy
+     * @return a new fix sized wrapper for native (64-bit address)
+     * memory with the contents copied from the given {@code bytes} array
+     */
+
+    static VanillaBytes allocateDirect( byte[] bytes) throws IllegalArgumentException {
+        VanillaBytes<Void> result = allocateDirect(bytes.length);
+        try {
+            result.write(bytes);
+        } catch (BufferOverflowException e) {
+            throw new AssertionError(e);
+        }
+        return result;
+    }
+
+
+    static Bytes fromHexString( String s) {
+        return BytesInternal.fromHexString(s);
+    }
+
+    /**
+     * Code shared by String and StringBuffer to do searches. The
+     * source is the character array being searched, and the target
+     * is the string being searched for.
+     *
+     * @param source    the read bytes being searched.
+     * @param target    the read bytes being searched for.
+     * @param fromIndex the index to begin searching from,
+     * @return the index of where the text was found.
+     */
+    static int indexOf( BytesStore source,  BytesStore target, int fromIndex) {
+
+        long sourceOffset = source.readPosition();
+        long targetOffset = target.readPosition();
+        long sourceCount = source.readRemaining();
+        long targetCount = target.readRemaining();
+
+        if (fromIndex >= sourceCount) {
+            return Math.toIntExact(targetCount == 0 ? sourceCount : -1);
+        }
+        if (fromIndex < 0) {
+            fromIndex = 0;
+        }
+        if (targetCount == 0) {
+            return fromIndex;
+        }
+
+        byte firstByte = target.readByte(targetOffset);
+        long max = sourceOffset + (sourceCount - targetCount);
+
+        for (long i = sourceOffset + fromIndex; i <= max; i++) {
+            /* Look for first character. */
+            if (source.readByte(i) != firstByte) {
+                while (++i <= max && source.readByte(i) != firstByte) ;
+            }
+
+            /* Found first character, now look at the rest of v2 */
+            if (i <= max) {
+                long j = i + 1;
+                long end = j + targetCount - 1;
+                for (long k = targetOffset + 1; j < end && source.readByte(j) == target.readByte(k); j++, k++) {
+                }
+
+                if (j == end) {
+                    /* Found whole string. */
+                    return Math.toIntExact(i - sourceOffset);
+                }
+            }
+        }
+        return -1;
+    }
+
+    /**
+     * Return a Bytes which is optionally unchecked.  This allows bounds checks to be turned off.
+     * Note: this means that the result is no longer elastic, even if <code>this</code> is elastic.
+     *
+     * @param unchecked if true, minimal bounds checks will be performed.
+     * @return Bytes without bounds checking.
+     * @throws IllegalStateException if the underlying BytesStore has been released
+     */
+
+    default Bytes<Underlying> unchecked(boolean unchecked) throws IllegalStateException {
+        if (unchecked) {
+            if (isElastic())
+                BytesUtil.WarnUncheckedElasticBytes.warn();
+            Bytes<Underlying> underlyingBytes = start() == 0 && bytesStore().isDirectMemory() ?
+                    new UncheckedNativeBytes<>(this) :
+                    new UncheckedBytes<>(this);
+            release(INIT);
+            return underlyingBytes;
+        }
+        return this;
+    }
+
+    default boolean unchecked() {
+        return false;
+    }
+
+    /**
+     * @inheritDoc <P>
+     * If this Bytes {@link #isElastic()} the {@link #safeLimit()} can be
+     * lower than the point it can safely write.
+     */
+    @Override
+    default long safeLimit() {
+        return bytesStore().safeLimit();
+    }
+
+    @Override
+    default boolean isClear() {
+        return start() == readPosition() && writeLimit() == capacity();
+    }
+
+    /**
+     * @inheritDoc <P>
+     * If this Bytes {@link #isElastic()} the {@link #realCapacity()} can be
+     * lower than the virtual {@link #capacity()}.
+     */
+    @Override
+    default long realCapacity() {
+        return BytesStore.super.realCapacity();
+    }
+
+    /**
+     * @return a copy of this Bytes from position() to limit().
+     */
+    @Override
+    BytesStore<Bytes<Underlying>, Underlying> copy();
+
+
+    default String toHexString() {
+        return toHexString(1024);
+    }
+
+    /**
+     * display the hex data of {@link Bytes} from the position() to the limit()
+     *
+     * @param maxLength limit the number of bytes to be dumped.
+     * @return hex representation of the buffer, from example [0D ,OA, FF]
+     */
+
+    default String toHexString(long maxLength) {
+        return toHexString(readPosition(), maxLength);
+    }
+
+    /**
+     * display the hex data of {@link Bytes} from the position() to the limit()
+     *
+     * @param maxLength limit the number of bytes to be dumped.
+     * @return hex representation of the buffer, from example [0D ,OA, FF]
+     */
+
+    default String toHexString(long offset, long maxLength) {
+//        if (Jvm.isDebug() && Jvm.stackTraceEndsWith("Bytes", 3))
+//            return "Not Available";
+
+        long maxLength2 = Math.min(maxLength, readLimit() - offset);
+         String ret = BytesInternal.toHexString(this, offset, maxLength2);
+        return maxLength2 < readLimit() - offset ? ret + "... truncated" : ret;
+    }
+
+    /**
+     * Returns if this Bytes is elastic. I.e. it can resize when more data is written
+     * than it's {@link #realCapacity()}.
+     *
+     * @return if this Bytes is elastic
+     */
+    boolean isElastic();
+
+    /**
+     * grow the buffer if the buffer is elastic, if the buffer is not elastic and there is not
+     * enough capacity then this method will throws {@link java.nio.BufferOverflowException}
+     *
+     * @param size the capacity that you required
+     * @throws IllegalArgumentException if the buffer is not elastic and there is not enough space
+     */
+    default void ensureCapacity(long size) throws IllegalArgumentException {
+        if (size > capacity())
+            throw new IllegalArgumentException(isElastic() ? "todo" : "not elastic");
+    }
+
+    /**
+     * Creates a slice of the current Bytes based on its position() and limit().  As a sub-section
+     * of a Bytes it cannot be elastic.
+     *
+     * @return a slice of the existing Bytes where the start is moved to the position and the
+     * current limit determines the capacity.
+     * @throws IllegalStateException if the underlying BytesStore has been released
+     */
+
+    @Override
+    default Bytes<Underlying> bytesForRead() throws IllegalStateException {
+        return isClear() ? BytesStore.super.bytesForRead() : new SubBytes<>(this, readPosition(), readLimit() + start());
+    }
+
+    /**
+     * @return the ByteStore this Bytes wraps.
+     */
+    @Override
+
+    BytesStore bytesStore();
+
+    default boolean isEqual(String s) {
+        return StringUtils.isEqual(this, s);
+    }
+
+    /**
+     * Compact these Bytes by moving the readPosition to the start.
+     *
+     * @return this
+     */
+
+    Bytes<Underlying> compact();
+
+    /**
+     * copy bytes from one ByteStore to another
+     *
+     * @param store to copy to
+     * @return the number of bytes copied.
+     */
+    @Override
+    default long copyTo( BytesStore store) {
+        return BytesStore.super.copyTo(store);
+    }
+
+    @Override
+    default void copyTo( OutputStream out) throws IOException {
+        BytesStore.super.copyTo(out);
+    }
+
+    @Override
+    default boolean sharedMemory() {
+        return bytesStore().sharedMemory();
+    }
+
+    /**
+     * will unwrite from the offset upto the current write position of the destination bytes
+     *
+     * @param fromOffset the offset from the target byytes
+     * @param count      the number of bytes to un-write
+     */
+    default void unwrite(long fromOffset, int count) {
+        long wp = writePosition();
+
+        if (wp < fromOffset)
+            return;
+
+        write(fromOffset, this, fromOffset + count, wp - fromOffset - count);
+        writeSkip(-count);
+    }
+
+
+    default BigDecimal readBigDecimal() {
+        return new BigDecimal(readBigInteger(), Maths.toUInt31(readStopBit()));
+    }
+
+
+    default BigInteger readBigInteger() {
+        int length = Maths.toUInt31(readStopBit());
+        if (length == 0)
+            if (lenient())
+                return BigInteger.ZERO;
+            else
+                throw new BufferUnderflowException();
+         byte[] bytes = new byte[length];
+        read(bytes);
+        return new BigInteger(bytes);
+    }
+
+    /**
+     * Returns the index within this bytes of the first occurrence of the
+     * specified sub-bytes.
+     * <p>
+     * <p>The returned index is the smallest value <i>k</i> for which:
+     * <blockquote><pre>
+     * this.startsWith(bytes, <i>k</i>)
+     * </pre></blockquote>
+     * If no such value of <i>k</i> exists, then {@code -1} is returned.
+     *
+     * @param source the sub-bytes to search for.
+     * @return the index of the first occurrence of the specified sub-bytes,
+     * or {@code -1} if there is no such occurrence.
+     */
+    default long indexOf( Bytes source) {
+        return indexOf(source, 0);
+    }
+
+    /**
+     * Returns the index within this bytes of the first occurrence of the
+     * specified subbytes.
+     * <p>
+     * <p>The returned index is the smallest value <i>k</i> for which:
+     * <blockquote><pre>
+     * this.startsWith(bytes, <i>k</i>)
+     * </pre></blockquote>
+     * If no such value of <i>k</i> exists, then {@code -1} is returned.
+     *
+     * @param source    the sub-bytes to search for.
+     * @param fromIndex start the seach from this offset
+     * @return the index of the first occurrence of the specified sub-bytes,
+     * or {@code -1} if there is no such occurrence.
+     */
+    default int indexOf( BytesStore source, int fromIndex) {
+        return indexOf(this, source, fromIndex);
+    }
+
+    @Deprecated(/* to be removed in x.22 */)
+    default long indexOf( Bytes source, int fromIndex) {
+        return indexOf(this, source, fromIndex);
+    }
+
+    @Override
+
+    Bytes<Underlying> clear();
+
+    @Override
+    default boolean readWrite() {
+        return bytesStore().readWrite();
+    }
+
+    default void readWithLength(long length,  BytesOut<Underlying> bytesOut)
+            throws BufferUnderflowException, IORuntimeException {
+        if (length > readRemaining())
+            throw new BufferUnderflowException();
+        long limit0 = readLimit();
+        long limit = readPosition() + length;
+        boolean lenient = lenient();
+        try {
+            lenient(true);
+            readLimit(limit);
+            bytesOut.write(this);
+        } finally {
+            readLimit(limit0);
+            readPosition(limit);
+            lenient(lenient);
+        }
+    }
+
+    default <T extends ReadBytesMarshallable> T readMarshallableLength16(Class<T> tClass, T object) {
+        if (object == null) object = ObjectUtils.newInstance(tClass);
+        int length = readUnsignedShort();
+        long limit = readLimit();
+        long end = readPosition() + length;
+        boolean lenient = lenient();
+        try {
+            lenient(true);
+            readLimit(end);
+            object.readMarshallable(this);
+        } finally {
+            readPosition(end);
+            readLimit(limit);
+            lenient(lenient);
+        }
+        return object;
+    }
+
+    default void writeMarshallableLength16(WriteBytesMarshallable marshallable) {
+        long position = writePosition();
+        writeUnsignedShort(0);
+        marshallable.writeMarshallable(this);
+        long length = writePosition() - position - 2;
+        if (length >= 1 << 16)
+            throw new IllegalStateException("Marshallable " + marshallable.getClass() + " too long was " + length);
+        writeUnsignedShort(position, (int) length);
+    }
+
+    default Bytes write(final InputStream inputStream) throws IOException {
+        for (; ; ) {
+            int read;
+            read = inputStream.read();
+            if (read == -1)
+                break;
+            writeByte((byte) read);
+        }
+        return this;
+    }
+
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/BytesComment.java

@@ -0,0 +1,51 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+public interface BytesComment<B extends BytesComment<B>> {
+    /**
+     * Do these Bytes support saving comments
+     *
+     * @return true if comments are kept
+     */
+    default boolean retainsComments() {
+        return false;
+    }
+
+    /**
+     * Add comment as approriate for the toHexString format
+     *
+     * @param comment to add (or ignore)
+     * @return this
+     */
+    @SuppressWarnings("unchecked")
+    default B comment(CharSequence comment) {
+        return (B) this;
+    }
+
+    /**
+     * Adjust the indent for nested data
+     *
+     * @param n +1 indent in, -1 reduce indenting
+     * @return this.
+     */
+    @SuppressWarnings("unchecked")
+    default B indent(int n) {
+        return (B) this;
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/BytesConsumer.java

@@ -0,0 +1,36 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import java.nio.BufferOverflowException;
+
+@FunctionalInterface
+public interface BytesConsumer {
+
+    /**
+     * Retrieves and removes the head of this queue, or returns {@code true} if this queue is
+     * empty.
+     *
+     * @param bytes to read into
+     * @return false if this queue is empty
+     */
+    @SuppressWarnings("rawtypes")
+    boolean read(BytesOut bytes) throws BufferOverflowException;
+
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/BytesContext.java

@@ -0,0 +1,42 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.io.Closeable;
+
+public interface BytesContext extends Closeable {
+
+    /**
+     * @return a bytes to write to
+     */
+    @SuppressWarnings("rawtypes")
+    Bytes bytes();
+
+    /**
+     * @return the key to be written to
+     */
+    int key();
+
+    @Override
+    default boolean isClosed() {
+        throw new UnsupportedOperationException("todo");
+    }
+
+    default void rollbackOnClose() {
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/BytesIn.java

@@ -0,0 +1,76 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ *  Licensed under the Apache License, Version 2.0 (the "License");
+ *  you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.util.ObjectUtils;
+import org.jetbrains.annotations.NotNull;
+
+@SuppressWarnings({"rawtypes", "unchecked"})
+public interface BytesIn<Underlying> extends
+        RandomDataInput,
+        ByteStringParser<Bytes<Underlying>> {
+    /**
+     * Reads messages from this tails as methods.  It returns a BooleanSupplier which returns
+     *
+     * @param objects which implement the methods serialized to the file.
+     * @return a reader which will read one Excerpt at a time
+     */
+
+    default MethodReader bytesMethodReader(Object... objects) {
+        return new BytesMethodReaderBuilder(this).build(objects);
+    }
+
+
+    default BytesMethodReaderBuilder bytesMethodReaderBuilder() {
+        return new BytesMethodReaderBuilder(this);
+    }
+
+    <T extends ReadBytesMarshallable> T readMarshallableLength16(Class<T> tClass, T object);
+
+    default <T> T readObject(Class<T> componentType0) {
+        Class<T> componentType = ObjectUtils.implementationToUse(componentType0);
+        if (BytesMarshallable.class.isAssignableFrom(componentType)) {
+            BytesMarshallable bm = (BytesMarshallable) ObjectUtils.newInstance(componentType);
+            bm.readMarshallable(this);
+            return (T) bm;
+        }
+        if (Enum.class.isAssignableFrom(componentType)) {
+            return (T) readEnum((Class) componentType);
+        }
+        switch (componentType.getName()) {
+            case "java.lang.String":
+                return (T) readUtf8();
+            case "java.lang.Double":
+                return (T) (Double) readDouble();
+            case "java.lang.Long":
+                return (T) (Long) readLong();
+            case "java.lang.Integer":
+                return (T) (Integer) readInt();
+
+            default:
+                if (BytesMarshallable.class.isAssignableFrom(componentType)) {
+                    BytesMarshallable bm = (BytesMarshallable) ObjectUtils.newInstance(componentType);
+                    bm.readMarshallable(this);
+                    return (T) bm;
+                }
+                throw new UnsupportedOperationException("Unsupported " + componentType);
+        }
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/BytesMarshallable.java

@@ -0,0 +1,48 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.annotation.DontChain;
+import net.openhft.chronicle.core.io.IORuntimeException;
+
+/**
+ * An object which can be read or written directly to Bytes in a streaming manner.
+ */
+@DontChain
+public interface BytesMarshallable extends ReadBytesMarshallable, WriteBytesMarshallable {
+    @Override
+    @SuppressWarnings("rawtypes")
+    default void readMarshallable(BytesIn bytes) throws IORuntimeException {
+        BytesUtil.readMarshallable(this, bytes);
+    }
+
+    @SuppressWarnings("rawtypes")
+    @Override
+    default void writeMarshallable(BytesOut bytes) {
+        BytesUtil.writeMarshallable(this, bytes);
+    }
+
+    default String $toString() {
+        HexDumpBytes bytes = new HexDumpBytes();
+        writeMarshallable(bytes);
+        String s = "# " + getClass().getName() + "\n" + bytes.toHexString();
+        bytes.releaseLast();
+        return s;
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/BytesMarshaller.java

@@ -0,0 +1,759 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.ClassLocal;
+import net.openhft.chronicle.core.Jvm;
+import net.openhft.chronicle.core.Maths;
+import net.openhft.chronicle.core.io.IORuntimeException;
+import net.openhft.chronicle.core.util.ObjectUtils;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+import java.lang.reflect.*;
+import java.util.*;
+import java.util.function.Supplier;
+
+@SuppressWarnings({"rawtypes", "unchecked"})
+public class BytesMarshaller<T> {
+    public static final ClassLocal<BytesMarshaller> BYTES_MARSHALLER_CL
+            = ClassLocal.withInitial(BytesMarshaller::new);
+    private final FieldAccess[] fields;
+
+    public BytesMarshaller( Class<T> tClass) {
+         Map<String, Field> map = new LinkedHashMap<>();
+        getAllField(tClass, map);
+        fields = map.values().stream()
+                .map(FieldAccess::create).toArray(FieldAccess[]::new);
+    }
+
+    public static void getAllField( Class clazz,  Map<String, Field> map) {
+        if (clazz != Object.class)
+            getAllField(clazz.getSuperclass(), map);
+        for ( Field field : clazz.getDeclaredFields()) {
+            if ((field.getModifiers() & (Modifier.STATIC | Modifier.TRANSIENT)) != 0)
+                continue;
+            Jvm.setAccessible(field);
+            map.put(field.getName(), field);
+        }
+    }
+
+    public void readMarshallable(ReadBytesMarshallable t, BytesIn in) {
+        for ( FieldAccess field : fields) {
+            field.read(t, in);
+        }
+    }
+
+    public void writeMarshallable(WriteBytesMarshallable t, BytesOut out) {
+        out.indent(+1);
+        for ( FieldAccess field : fields) {
+            field.write(t, out);
+        }
+        out.indent(-1);
+    }
+
+    static abstract class FieldAccess<T> {
+        final Field field;
+
+        FieldAccess(Field field) {
+            this.field = field;
+        }
+
+
+        public static Object create( Field field) {
+            Class<?> type = field.getType();
+            switch (type.getName()) {
+                case "boolean":
+                    return new BooleanFieldAccess(field);
+                case "byte":
+                    return new ByteFieldAccess(field);
+                case "char":
+                    return new CharFieldAccess(field);
+                case "short":
+                    return new ShortFieldAccess(field);
+                case "int":
+                    return new IntegerFieldAccess(field);
+                case "float":
+                    return new FloatFieldAccess(field);
+                case "long":
+                    return new LongFieldAccess(field);
+                case "double":
+                    return new DoubleFieldAccess(field);
+                default:
+                    if (type.isArray()) {
+                        if (type.getComponentType().isPrimitive()) {
+                            if (type == byte[].class)
+                                return new ByteArrayFieldAccess(field);
+                            if (type == int[].class)
+                                return new IntArrayFieldAccess(field);
+                            if (type == float[].class)
+                                return new FloatArrayFieldAccess(field);
+                            if (type == long[].class)
+                                return new LongArrayFieldAccess(field);
+                            if (type == double[].class)
+                                return new DoubleArrayFieldAccess(field);
+                            throw new UnsupportedOperationException("TODO " + field.getType());
+                        }
+                        return new ObjectArrayFieldAccess(field);
+                    }
+                    if (Collection.class.isAssignableFrom(type))
+                        return new CollectionFieldAccess(field);
+                    if (Map.class.isAssignableFrom(type))
+                        return new MapFieldAccess(field);
+                    if (BytesStore.class.isAssignableFrom(type))
+                        return new BytesFieldAccess(field);
+                    if (BytesMarshallable.class.isAssignableFrom(type))
+                        return new BytesMarshallableFieldAccess(field);
+                    return new ScalarFieldAccess(field);
+            }
+        }
+
+
+        static Class extractClass(Type type0) {
+            if (type0 instanceof Class)
+                return (Class) type0;
+            else if (type0 instanceof ParameterizedType)
+                return (Class) ((ParameterizedType) type0).getRawType();
+            else
+                return Object.class;
+        }
+
+
+        @Override
+        public String toString() {
+            return getClass().getSimpleName() + "{" +
+                    "field=" + field +
+                    '}';
+        }
+
+        void write(Object o, BytesOut write) {
+            try {
+                write.comment(field.getName());
+                getValue(o, write);
+            } catch (IllegalAccessException iae) {
+                throw new AssertionError(iae);
+            }
+        }
+
+        protected abstract void getValue(Object o, BytesOut write) throws IllegalAccessException;
+
+        void read(Object o, BytesIn read) {
+            try {
+                setValue(o, read);
+            } catch (IllegalAccessException iae) {
+                throw new AssertionError(iae);
+            } catch (IORuntimeException e) {
+                throw Jvm.rethrow(e);
+            }
+        }
+
+        protected abstract void setValue(Object o, BytesIn read) throws IllegalAccessException, IORuntimeException;
+
+
+        protected Supplier<Map> newInstance( Class type) {
+            try {
+                return (Supplier<Map>) type.newInstance();
+            } catch (IllegalAccessException e) {
+                throw new AssertionError(e);
+            } catch (InstantiationException e) {
+                throw Jvm.rethrow(e);
+            }
+        }
+    }
+
+    static class ScalarFieldAccess extends FieldAccess<Object> {
+        public ScalarFieldAccess(Field field) {
+            super(field);
+        }
+
+        @Override
+        protected void getValue(Object o,  BytesOut write) throws IllegalAccessException {
+            Object o2 = field.get(o);
+             String s = o2 == null ? null : o2.toString();
+            write.writeUtf8(s);
+        }
+
+        @Override
+        protected void setValue(Object o,  BytesIn read) throws IllegalAccessException, IORuntimeException {
+             String s = read.readUtf8();
+            field.set(o, ObjectUtils.convertTo(field.getType(), s));
+        }
+    }
+
+    static class BytesMarshallableFieldAccess extends FieldAccess<Object> {
+        public BytesMarshallableFieldAccess(Field field) {
+            super(field);
+        }
+
+        @Override
+        protected void getValue(Object o, BytesOut write) throws IllegalAccessException {
+             BytesMarshallable o2 = (BytesMarshallable) field.get(o);
+            assert o2 != null;
+            o2.writeMarshallable(write);
+        }
+
+        @Override
+        protected void setValue(Object o, BytesIn read) throws IllegalAccessException, IORuntimeException {
+             BytesMarshallable o2 = (BytesMarshallable) field.get(o);
+            if (!field.getType().isInstance(o2))
+                field.set(o, o2 = (BytesMarshallable) ObjectUtils.newInstance((Class) field.getType()));
+
+            o2.readMarshallable(read);
+        }
+    }
+
+    static class BytesFieldAccess extends FieldAccess<Bytes> {
+        public BytesFieldAccess(Field field) {
+            super(field);
+        }
+
+        @Override
+        protected void getValue(Object o,  BytesOut write) throws IllegalAccessException {
+             BytesStore bytes = (BytesStore) field.get(o);
+            if (bytes == null) {
+                write.writeStopBit(-1);
+                return;
+            }
+            long offset = bytes.readPosition();
+            long length = bytes.readRemaining();
+            write.writeStopBit(length);
+            write.write(bytes, offset, length);
+        }
+
+        @Override
+        protected void setValue(Object o,  BytesIn read) throws IllegalAccessException, IORuntimeException {
+             Bytes bytes = (Bytes) field.get(o);
+            long stopBit = read.readStopBit();
+            if (stopBit == -1) {
+                if (bytes != null)
+                    bytes.releaseLast();
+                field.set(o, null);
+                return;
+            }
+            int length = Maths.toUInt31(stopBit);
+             Bytes bs;
+            if (bytes == null) {
+                bs = Bytes.allocateElasticOnHeap(length);
+                field.set(o, bs);
+            } else {
+                bs = bytes;
+            }
+            bs.clear();
+            read.read(bs, length);
+            bs.readLimit(length);
+        }
+    }
+
+/*
+    static class ArrayFieldAccess extends FieldAccess {
+        private final Class componentType;
+
+        public ArrayFieldAccess( Field field) {
+            super(field);
+            componentType = field.getType().getComponentType();
+        }
+
+        @Override
+        protected void getValue(Object o, BytesOut write) throws IllegalAccessException {
+            throw new UnsupportedOperationException("TODO");
+        }
+
+        @Override
+        protected void setValue(Object o, BytesIn read) throws IllegalAccessException {
+            throw new UnsupportedOperationException("TODO");
+        }
+    }
+*/
+
+    static class ObjectArrayFieldAccess extends FieldAccess {
+        Class componentType;
+
+        public ObjectArrayFieldAccess(Field field) {
+            super(field);
+            componentType = field.getType().getComponentType();
+        }
+
+        @Override
+        protected void getValue(Object o, BytesOut write) throws IllegalAccessException {
+            Object[] c = (Object[]) field.get(o);
+            if (c == null) {
+                write.writeStopBit(-1);
+                return;
+            }
+            int size = c.length;
+            write.writeStopBit(size);
+            if (size == 0)
+                return;
+            for (int i = 0; i < size; i++)
+                write.writeObject(componentType, c[i]);
+        }
+
+        @Override
+        protected void setValue(Object o, BytesIn read) throws IllegalAccessException {
+            Object[] c = (Object[]) field.get(o);
+            int length = Maths.toInt32(read.readStopBit());
+            if (length < 0) {
+                if (c != null)
+                    field.set(o, null);
+                return;
+            }
+            if (c == null)
+                field.set(o, c = (Object[]) Array.newInstance(field.getType().getComponentType(), length));
+            else if (c.length != length)
+                field.set(o, c = Arrays.copyOf(c, length));
+            for (int i = 0; i < length; i++) {
+                Object o2 = c[i];
+                if (o2 instanceof BytesMarshallable)
+                    ((BytesMarshallable) o2).readMarshallable(read);
+                else
+                    c[i] = read.readObject(componentType);
+            }
+        }
+    }
+
+    static class CollectionFieldAccess extends FieldAccess {
+        final Supplier<Collection> collectionSupplier;
+
+        private final Class componentType;
+        private final Class<?> type;
+
+        public CollectionFieldAccess( Field field) {
+            super(field);
+            type = field.getType();
+            if (type == List.class || type == Collection.class)
+                collectionSupplier = ArrayList::new;
+            else if (type == SortedSet.class || type == NavigableSet.class)
+                collectionSupplier = TreeSet::new;
+            else if (type == Set.class)
+                collectionSupplier = LinkedHashSet::new;
+            else
+                collectionSupplier = newInstance(type);
+            Type genericType = field.getGenericType();
+            if (genericType instanceof ParameterizedType) {
+                 ParameterizedType pType = (ParameterizedType) genericType;
+                Type type0 = pType.getActualTypeArguments()[0];
+                componentType = extractClass(type0);
+            } else {
+                componentType = Object.class;
+            }
+        }
+
+        @Override
+        protected void getValue(Object o, BytesOut write) throws IllegalAccessException {
+            Collection c = (Collection) field.get(o);
+            if (c == null) {
+                write.writeStopBit(-1);
+                return;
+            }
+            write.writeStopBit(c.size());
+            if (c.isEmpty())
+                return;
+            if (c instanceof RandomAccess && c instanceof List) {
+                List l = (List) c;
+                for (int i = 0, size = l.size(); i < size; i++)
+                    write.writeObject(componentType, l.get(i));
+            } else {
+                for (Object o2 : c) {
+                    write.writeObject(componentType, o2);
+                }
+            }
+        }
+
+        @Override
+        protected void setValue(Object o, BytesIn read) throws IllegalAccessException {
+            Collection c = (Collection) field.get(o);
+            int length = Maths.toInt32(read.readStopBit());
+            if (length < 0) {
+                if (c != null)
+                    field.set(o, null);
+                return;
+            }
+            c.clear();
+            for (int i = 0; i < length; i++)
+                c.add(read.readObject(componentType));
+        }
+    }
+
+    static class MapFieldAccess extends FieldAccess {
+        final Supplier<Map> collectionSupplier;
+        private final Class<?> type;
+
+        private final Class keyType;
+
+        private final Class valueType;
+
+        public MapFieldAccess( Field field) {
+            super(field);
+            type = field.getType();
+            if (type == Map.class)
+                collectionSupplier = LinkedHashMap::new;
+            else if (type == SortedMap.class || type == NavigableMap.class)
+                collectionSupplier = TreeMap::new;
+            else
+                collectionSupplier = newInstance(type);
+            Type genericType = field.getGenericType();
+            if (genericType instanceof ParameterizedType) {
+                 ParameterizedType pType = (ParameterizedType) genericType;
+                Type[] actualTypeArguments = pType.getActualTypeArguments();
+                keyType = extractClass(actualTypeArguments[0]);
+                valueType = extractClass(actualTypeArguments[1]);
+
+            } else {
+                keyType = Object.class;
+                valueType = Object.class;
+            }
+        }
+
+        @Override
+        protected void getValue(Object o, BytesOut write) throws IllegalAccessException {
+            Map<?, ?> m = (Map) field.get(o);
+            if (m == null) {
+                write.writeStopBit(-1);
+                return;
+            }
+            write.writeStopBit(m.size());
+            for (Map.Entry<?, ?> entry : m.entrySet()) {
+                write.writeObject(keyType, entry.getKey());
+                write.writeObject(valueType, entry.getValue());
+            }
+        }
+
+        @Override
+        protected void setValue(Object o, BytesIn read) throws IllegalAccessException {
+            Map m = (Map) field.get(o);
+            long length = read.readStopBit();
+            if (length < 0) {
+                if (m != null)
+                    field.set(o, null);
+                return;
+            }
+            if (m == null) {
+                field.set(o, m = new LinkedHashMap<>());
+            } else {
+                m.clear();
+            }
+            for (int i = 0; i < length; i++) {
+                m.put(read.readObject(keyType), read.readObject(valueType));
+            }
+        }
+    }
+
+    static class BooleanFieldAccess extends FieldAccess {
+        public BooleanFieldAccess(Field field) {
+            super(field);
+        }
+
+        @Override
+        protected void getValue(Object o,  BytesOut write) throws IllegalAccessException {
+            write.writeBoolean(field.getBoolean(o));
+        }
+
+        @Override
+        protected void setValue(Object o,  BytesIn read) throws IllegalAccessException {
+            field.setBoolean(o, read.readBoolean());
+        }
+    }
+
+    static class ByteFieldAccess extends FieldAccess {
+        public ByteFieldAccess(Field field) {
+            super(field);
+        }
+
+        @Override
+        protected void getValue(Object o,  BytesOut write) throws IllegalAccessException {
+            write.writeByte(field.getByte(o));
+        }
+
+        @Override
+        protected void setValue(Object o,  BytesIn read) throws IllegalAccessException {
+            field.setByte(o, read.readByte());
+        }
+    }
+
+    static class ByteArrayFieldAccess extends FieldAccess {
+        public ByteArrayFieldAccess(Field field) {
+            super(field);
+        }
+
+        @Override
+        protected void getValue(Object o,  BytesOut write) throws IllegalAccessException {
+            byte[] array = (byte[]) field.get(o);
+            if (array == null) {
+                write.writeInt(~0);
+            } else {
+                write.writeInt(array.length);
+                for (int i = 0; i < array.length; i++)
+                    write.writeByte(array[i]);
+            }
+        }
+
+        @Override
+        protected void setValue(Object o,  BytesIn read) throws IllegalAccessException {
+            int len = read.readInt();
+            if (len == ~0) {
+                field.set(o, null);
+            } else if (len >= 0) {
+                byte[] array = (byte[]) field.get(o);
+                if (array == null || array.length != len) {
+                    array = new byte[len];
+                    field.set(o, array);
+                }
+                for (int i = 0; i < len; i++)
+                    array[i] = read.readByte();
+            }
+        }
+    }
+
+    static class CharFieldAccess extends FieldAccess {
+        public CharFieldAccess(Field field) {
+            super(field);
+        }
+
+        @Override
+        protected void getValue(Object o,  BytesOut write) throws IllegalAccessException {
+            char aChar = field.getChar(o);
+            if (aChar >= 65536 - 127)
+                write.writeStopBit(aChar - 65536);
+            else
+                write.writeStopBit(aChar);
+        }
+
+        @Override
+        protected void setValue(Object o,  BytesIn read) throws IllegalAccessException {
+            field.setChar(o, read.readStopBitChar());
+        }
+    }
+
+    static class ShortFieldAccess extends FieldAccess {
+        public ShortFieldAccess(Field field) {
+            super(field);
+        }
+
+        @Override
+        protected void getValue(Object o,  BytesOut write) throws IllegalAccessException {
+            write.writeShort(field.getShort(o));
+        }
+
+        @Override
+        protected void setValue(Object o,  BytesIn read) throws IllegalAccessException {
+            field.setShort(o, read.readShort());
+        }
+    }
+
+    static class IntegerFieldAccess extends FieldAccess {
+        public IntegerFieldAccess(Field field) {
+            super(field);
+        }
+
+        @Override
+        protected void getValue(Object o,  BytesOut write) throws IllegalAccessException {
+            write.writeInt(field.getInt(o));
+        }
+
+        @Override
+        protected void setValue(Object o,  BytesIn read) throws IllegalAccessException {
+            field.setInt(o, read.readInt());
+        }
+    }
+
+    static class IntArrayFieldAccess extends FieldAccess {
+        public IntArrayFieldAccess(Field field) {
+            super(field);
+        }
+
+        @Override
+        protected void getValue(Object o,  BytesOut write) throws IllegalAccessException {
+            int[] array = (int[]) field.get(o);
+            if (array == null) {
+                write.writeInt(~0);
+            } else {
+                write.writeInt(array.length);
+                for (int i = 0; i < array.length; i++)
+                    write.writeInt(array[i]);
+            }
+        }
+
+        @Override
+        protected void setValue(Object o,  BytesIn read) throws IllegalAccessException {
+            int len = read.readInt();
+            if (len == ~0) {
+                field.set(o, null);
+            } else if (len >= 0) {
+                int[] array = (int[]) field.get(o);
+                if (array == null || array.length != len) {
+                    array = new int[len];
+                    field.set(o, array);
+                }
+                for (int i = 0; i < len; i++)
+                    array[i] = read.readInt();
+            }
+        }
+    }
+
+    static class FloatFieldAccess extends FieldAccess {
+        public FloatFieldAccess(Field field) {
+            super(field);
+        }
+
+        @Override
+        protected void getValue(Object o,  BytesOut write) throws IllegalAccessException {
+            write.writeFloat(field.getFloat(o));
+        }
+
+        @Override
+        protected void setValue(Object o,  BytesIn read) throws IllegalAccessException {
+            field.setFloat(o, read.readFloat());
+        }
+    }
+
+    static class FloatArrayFieldAccess extends FieldAccess {
+        public FloatArrayFieldAccess(Field field) {
+            super(field);
+        }
+
+        @Override
+        protected void getValue(Object o,  BytesOut write) throws IllegalAccessException {
+            float[] array = (float[]) field.get(o);
+            if (array == null) {
+                write.writeInt(~0);
+            } else {
+                write.writeInt(array.length);
+                for (int i = 0; i < array.length; i++)
+                    write.writeFloat(array[i]);
+            }
+        }
+
+        @Override
+        protected void setValue(Object o,  BytesIn read) throws IllegalAccessException {
+            int len = read.readInt();
+            if (len == ~0) {
+                field.set(o, null);
+            } else if (len >= 0) {
+                float[] array = (float[]) field.get(o);
+                if (array == null || array.length != len) {
+                    array = new float[len];
+                    field.set(o, array);
+                }
+                for (int i = 0; i < len; i++)
+                    array[i] = read.readFloat();
+            }
+        }
+    }
+
+    static class LongFieldAccess extends FieldAccess {
+        public LongFieldAccess(Field field) {
+            super(field);
+        }
+
+        @Override
+        protected void getValue(Object o,  BytesOut write) throws IllegalAccessException {
+            write.writeLong(field.getLong(o));
+        }
+
+        @Override
+        protected void setValue(Object o,  BytesIn read) throws IllegalAccessException {
+            field.setLong(o, read.readLong());
+        }
+    }
+
+    static class LongArrayFieldAccess extends FieldAccess {
+        public LongArrayFieldAccess(Field field) {
+            super(field);
+        }
+
+        @Override
+        protected void getValue(Object o,  BytesOut write) throws IllegalAccessException {
+            long[] array = (long[]) field.get(o);
+            if (array == null) {
+                write.writeInt(~0);
+            } else {
+                write.writeInt(array.length);
+                for (int i = 0; i < array.length; i++)
+                    write.writeLong(array[i]);
+            }
+        }
+
+        @Override
+        protected void setValue(Object o,  BytesIn read) throws IllegalAccessException {
+            int len = read.readInt();
+            if (len == ~0) {
+                field.set(o, null);
+            } else if (len >= 0) {
+                long[] array = (long[]) field.get(o);
+                if (array == null || array.length != len) {
+                    array = new long[len];
+                    field.set(o, array);
+                }
+                for (int i = 0; i < len; i++)
+                    array[i] = read.readLong();
+            }
+        }
+    }
+
+    static class DoubleFieldAccess extends FieldAccess {
+        public DoubleFieldAccess(Field field) {
+            super(field);
+        }
+
+        @Override
+        protected void getValue(Object o,  BytesOut write) throws IllegalAccessException {
+            write.writeDouble(field.getDouble(o));
+        }
+
+        @Override
+        protected void setValue(Object o,  BytesIn read) throws IllegalAccessException {
+            field.setDouble(o, read.readDouble());
+        }
+    }
+
+    static class DoubleArrayFieldAccess extends FieldAccess {
+        public DoubleArrayFieldAccess(Field field) {
+            super(field);
+        }
+
+        @Override
+        protected void getValue(Object o,  BytesOut write) throws IllegalAccessException {
+            double[] array = (double[]) field.get(o);
+            if (array == null) {
+                write.writeInt(~0);
+            } else {
+                write.writeInt(array.length);
+                for (int i = 0; i < array.length; i++)
+                    write.writeDouble(array[i]);
+            }
+        }
+
+        @Override
+        protected void setValue(Object o,  BytesIn read) throws IllegalAccessException {
+            int len = read.readInt();
+            if (len == ~0) {
+                field.set(o, null);
+            } else if (len >= 0) {
+                double[] array = (double[]) field.get(o);
+                if (array == null || array.length != len) {
+                    array = new double[len];
+                    field.set(o, array);
+                }
+                for (int i = 0; i < len; i++)
+                    array[i] = read.readDouble();
+            }
+        }
+    }
+
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/BytesMethodReader.java

@@ -0,0 +1,94 @@
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.Jvm;
+import net.openhft.chronicle.core.io.SimpleCloseable;
+import net.openhft.chronicle.core.util.InvocationTargetRuntimeException;
+import net.openhft.chronicle.core.util.ObjectUtils;
+
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.*;
+import java.util.function.Consumer;
+
+@SuppressWarnings("rawtypes")
+public class BytesMethodReader extends SimpleCloseable implements MethodReader {
+    private final BytesIn in;
+    private final BytesParselet defaultParselet;
+    private final List<Consumer<BytesIn>> methodEncoders = new ArrayList<>();
+    private final Map<Long, Consumer<BytesIn>> methodEncoderMap = new LinkedHashMap<>();
+
+    public BytesMethodReader(BytesIn in,
+                             BytesParselet defaultParselet,
+                             MethodEncoderLookup methodEncoderLookup,
+                             Object[] objects) {
+
+        this.in = in;
+        this.defaultParselet = defaultParselet;
+
+        for (Object object : objects) {
+            for (Method method : object.getClass().getMethods()) {
+                MethodEncoder encoder = methodEncoderLookup.apply(method);
+                if (encoder != null) {
+                    addEncoder(object, method, encoder);
+                }
+            }
+        }
+    }
+
+    private void addEncoder(Object object, Method method, MethodEncoder encoder) {
+        Jvm.setAccessible(method);
+        Class<?>[] parameterTypes = method.getParameterTypes();
+        int count = parameterTypes.length;
+        BytesMarshallable[][] array = new BytesMarshallable[1][count];
+        for (int i = 0; i < count; i++) {
+            array[0][i] = (BytesMarshallable) ObjectUtils.newInstance(parameterTypes[i]);
+        }
+        Consumer<BytesIn> reader = in -> {
+            array[0] = (BytesMarshallable[]) encoder.decode(array[0], in);
+            try {
+                method.invoke(object, array[0]);
+            } catch (IllegalAccessException | InvocationTargetException e) {
+                Jvm.warn().on(getClass(), "Exception calling " + method + " " + Arrays.toString(array[0]), e);
+            }
+        };
+        long messageId = encoder.messageId();
+        if (messageId >= 0 && messageId < 1000) {
+            while (methodEncoders.size() <= messageId)
+                methodEncoders.add(null);
+            methodEncoders.set((int) messageId, reader);
+        } else {
+            methodEncoderMap.put(messageId, reader);
+        }
+    }
+
+    @Override
+    public MethodReaderInterceptorReturns methodReaderInterceptorReturns() {
+        throw new UnsupportedOperationException();
+    }
+
+    public boolean readOne() throws InvocationTargetRuntimeException {
+        throwExceptionIfClosed();
+
+        if (in.readRemaining() < 1)
+            return false;
+        long messageId = in.readStopBit();
+        Consumer<BytesIn> consumer;
+        if (messageId >= 0 && messageId < methodEncoders.size())
+            consumer = methodEncoders.get((int) messageId);
+        else
+            consumer = methodEncoderMap.get(messageId);
+        if (consumer == null) {
+            defaultParselet.accept(messageId, in);
+        } else {
+            consumer.accept(in);
+        }
+
+        return true;
+    }
+
+    @Override
+    public MethodReader closeIn(boolean closeIn) {
+        return this;
+    }
+
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/BytesMethodReaderBuilder.java

@@ -0,0 +1,72 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+import org.jetbrains.annotations.NotNull;
+
+@SuppressWarnings("rawtypes")
+public class BytesMethodReaderBuilder implements MethodReaderBuilder {
+    private final BytesIn in;
+    private BytesParselet defaultParselet = createDefaultParselet();
+    private MethodEncoderLookup methodEncoderLookup = MethodEncoderLookup.BY_ANNOTATION;
+
+    public BytesMethodReaderBuilder(BytesIn in) {
+        this.in = in;
+    }
+
+
+    static BytesParselet createDefaultParselet() {
+        return (msg, in) -> {
+            Bytes bytes = (Bytes) in;
+            throw new IllegalArgumentException("Unknown message type " + msg + " " + bytes.toHexString());
+        };
+    }
+
+    @Override
+    public MethodReaderBuilder warnMissing(boolean warnMissing) {
+        // always true
+        return this;
+    }
+
+    public MethodEncoderLookup methodEncoderLookup() {
+        return methodEncoderLookup;
+    }
+
+    public BytesMethodReaderBuilder methodEncoderLookup(MethodEncoderLookup methodEncoderLookup) {
+        this.methodEncoderLookup = methodEncoderLookup;
+        return this;
+    }
+
+    public BytesParselet defaultParselet() {
+        return defaultParselet;
+    }
+
+    public BytesMethodReaderBuilder defaultParselet(BytesParselet defaultParselet) {
+        this.defaultParselet = defaultParselet;
+        return this;
+    }
+
+    @Override
+    public MethodReaderBuilder methodReaderInterceptorReturns(MethodReaderInterceptorReturns methodReaderInterceptorReturns) {
+        throw new UnsupportedOperationException();
+    }
+
+    public BytesMethodReader build(Object... objects) {
+        return new BytesMethodReader(in, defaultParselet, methodEncoderLookup, objects);
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/BytesMethodWriterInvocationHandler.java

@@ -0,0 +1,27 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.io.Closeable;
+
+import java.lang.reflect.InvocationHandler;
+
+public interface BytesMethodWriterInvocationHandler extends InvocationHandler {
+
+    void onClose(Closeable closeable);
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/BytesOut.java

@@ -0,0 +1,96 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ *  Licensed under the Apache License, Version 2.0 (the "License");
+ *  you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.util.ObjectUtils;
+import org.jetbrains.annotations.NotNull;
+
+import java.lang.reflect.Method;
+import java.lang.reflect.Proxy;
+import java.util.function.Function;
+
+@SuppressWarnings({"rawtypes", "unchecked"})
+public interface BytesOut<Underlying> extends
+        ByteStringAppender<Bytes<Underlying>>,
+        BytesPrepender<Bytes<Underlying>>,
+        BytesComment<BytesOut<Underlying>> {
+
+    /**
+     * Proxy an interface so each message called is written to a file for replay.
+     *
+     * @param tClass     primary interface
+     * @param additional any additional interfaces
+     * @return a proxy which implements the primary interface (additional interfaces have to be
+     * cast)
+     */
+
+    default <T> T bytesMethodWriter( Class<T> tClass, Class... additional) {
+        Class[] interfaces = ObjectUtils.addAll(tClass, additional);
+
+        //noinspection unchecked
+        return (T) Proxy.newProxyInstance(tClass.getClassLoader(), interfaces,
+                new BinaryBytesMethodWriterInvocationHandler(MethodEncoderLookup.BY_ANNOTATION, this));
+    }
+
+    void writeMarshallableLength16(WriteBytesMarshallable marshallable);
+
+    /**
+     * Write a limit set of writeObject types.
+     *
+     * @param componentType expected.
+     * @param obj           of componentType
+     */
+    default void writeObject(Class componentType, Object obj) {
+        if (!componentType.isInstance(obj))
+            throw new IllegalArgumentException("Cannot serialize " + obj.getClass() + " as an " + componentType);
+        if (obj instanceof BytesMarshallable) {
+            ((BytesMarshallable) obj).writeMarshallable(this);
+            return;
+        }
+        if (obj instanceof Enum) {
+            writeEnum((Enum) obj);
+            return;
+        }
+        if (obj instanceof BytesStore) {
+            BytesStore bs = (BytesStore) obj;
+            writeStopBit(bs.readRemaining());
+            write(bs);
+            return;
+        }
+        switch (componentType.getName()) {
+            case "java.lang.String":
+                writeUtf8((String) obj);
+                return;
+            case "java.lang.Double":
+                writeDouble((Double) obj);
+                return;
+            case "java.lang.Long":
+                writeLong((Long) obj);
+                return;
+            case "java.lang.Integer":
+                writeInt((Integer) obj);
+                return;
+
+            default:
+                throw new UnsupportedOperationException("Not supported " + componentType);
+        }
+    }
+
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/BytesParselet.java

@@ -0,0 +1,24 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+@FunctionalInterface
+public interface BytesParselet {
+    @SuppressWarnings("rawtypes")
+    void accept(long messageType, BytesIn in);
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/BytesPrepender.java

@@ -0,0 +1,109 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.io.IORuntimeException;
+import org.jetbrains.annotations.NotNull;
+
+import java.nio.BufferOverflowException;
+import java.nio.BufferUnderflowException;
+
+public interface BytesPrepender<B extends BytesPrepender<B>> {
+
+    /**
+     * Clear a buffer, with a given padding to allow for prepending later. clearAndPad(0) is the same as clear()
+     *
+     * @param length to pad
+     * @return this
+     * @throws BufferOverflowException if the length &gt; capacity() - start()
+     */
+
+    B clearAndPad(long length) throws BufferOverflowException;
+
+    /**
+     * Prepends a long in decimal, this method moves the readPosition() backwards.
+     * <p>Note: it moves the readPosition not the writePosition / readLimit</p>
+     *
+     * @param value to prepend as text.
+     * @return this
+     * @throws BufferUnderflowException if the capacity of the underlying buffer was exceeded
+     * @throws IORuntimeException       if an error occurred while attempting to resize the underlying buffer
+     */
+    @SuppressWarnings("unchecked")
+
+    default B prepend(long value) throws BufferOverflowException {
+        BytesInternal.prepend(this, value);
+        return (B) this;
+    }
+
+    /**
+     * Write backward in binary a byte
+     * <p>Note: it moves the readPosition not the writePosition / readLimit</p>
+     *
+     * @param bytes to prepend to.
+     */
+
+    B prewrite(byte[] bytes) throws BufferOverflowException;
+
+    /**
+     * Write backward in binary a byte
+     * <p>Note: it moves the readPosition not the writePosition / readLimit</p>
+     *
+     * @param bytes to prepend to.
+     */
+    @SuppressWarnings("rawtypes")
+
+    B prewrite(BytesStore bytes) throws BufferOverflowException;
+
+    /**
+     * Write backward in binary a byte
+     * <p>Note: it moves the readPosition not the writePosition / readLimit</p>
+     *
+     * @param b byte to prepend to.
+     */
+
+    B prewriteByte(byte b) throws BufferOverflowException;
+
+    /**
+     * Write backward in binary a 2 byte int
+     * <p>Note: it moves the readPosition not the writePosition / readLimit</p>
+     *
+     * @param i short to prepend to.
+     */
+
+    B prewriteShort(short i) throws BufferOverflowException;
+
+    /**
+     * Write backward in binary a 4 byte int
+     * <p>Note: it moves the readPosition not the writePosition / readLimit</p>
+     *
+     * @param i integer to prepend to.
+     */
+
+    B prewriteInt(int i) throws BufferOverflowException;
+
+    /**
+     * Write backward in binary an 8 byte long
+     * <p>Note: it moves the readPosition not the writePosition / readLimit</p>
+     *
+     * @param l long to prepend to.
+     */
+
+    B prewriteLong(long l) throws BufferOverflowException;
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/BytesRingBuffer.java

@@ -0,0 +1,115 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.Jvm;
+import net.openhft.chronicle.core.io.Closeable;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.Constructor;
+import java.lang.reflect.Method;
+import java.nio.BufferOverflowException;
+
+@SuppressWarnings({"rawtypes", "unchecked"})
+public interface BytesRingBuffer extends BytesRingBufferStats, BytesConsumer, Closeable {
+
+    Logger LOG = LoggerFactory.getLogger(BytesRingBuffer.class);
+
+
+    static BytesRingBuffer newInstance( NativeBytesStore<Void> bytesStore) {
+        return newInstance(bytesStore, 1);
+    }
+
+
+    static MultiReaderBytesRingBuffer newInstance(
+             NativeBytesStore<Void> bytesStore,
+            int numReaders) {
+        try {
+             final Class<MultiReaderBytesRingBuffer> aClass = clazz();
+            final Constructor<MultiReaderBytesRingBuffer> constructor = aClass
+                    .getDeclaredConstructor(BytesStore.class, int.class);
+            return constructor.newInstance(bytesStore, numReaders);
+
+        } catch (Exception e) {
+            LOG.error("This is a a commercial feature, please contact " +
+                    "sales@chronicle.software to unlock this feature.");
+
+            throw Jvm.rethrow(e);
+        }
+    }
+
+
+    static Class<MultiReaderBytesRingBuffer> clazz() throws ClassNotFoundException {
+        //noinspection AccessStaticViaInstance
+        return (Class<MultiReaderBytesRingBuffer>) Class.forName(
+                "software.chronicle.enterprise.ring.EnterpriseRingBuffer");
+    }
+
+    static long sizeFor(long capacity) {
+        return sizeFor(capacity, 1);
+    }
+
+    static long sizeFor(long capacity, int numReaders) {
+        try {
+            //noinspection AccessStaticViaInstance
+            final Method sizeFor = Class.forName(
+                    "software.chronicle.enterprise.queue.ChronicleRingBuffer").getMethod("sizeFor", long.class, int.class);
+            return (long) sizeFor.invoke(null, capacity, numReaders);
+
+        } catch (Exception e) {
+            LOG.error("This is a a commercial feature, please contact " +
+                    "sales@chronicle.software to unlock this feature.");
+
+            throw Jvm.rethrow(e);
+        }
+    }
+
+    /**
+     * clears the ring buffer but moving the read position to the write position
+     */
+    void clear();
+
+    /**
+     * Inserts the specified element at the tail of this queue if it is possible to do so
+     * immediately without exceeding the queue's capacity,
+     *
+     * @param bytes0 the {@code bytes0} that you wish to add to the ring buffer
+     * @return returning {@code true} upon success and {@code false} if this queue is full.
+     */
+    boolean offer( BytesStore bytes0);
+
+    /**
+     * Retrieves and removes the head of this queue, or returns {@code null} if this queue is
+     * empty.
+     *
+     * @param using Bytes to read into.
+     * @return false if this queue is empty, or a populated buffer if the element was retried
+     * @throws BufferOverflowException is the {@code using} buffer is not large enough
+     */
+    @Override
+    boolean read( BytesOut using);
+
+    long readRemaining();
+
+    boolean isEmpty();
+
+    BytesStore bytesStore();
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/BytesRingBufferStats.java

@@ -0,0 +1,44 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import java.util.List;
+
+public interface BytesRingBufferStats {
+    /**
+     * each time the ring is read, this logs the number of bytes in the write buffer, calling this
+     * method resets these statistics,
+     *
+     * @return Long.MAX_VALUE if no read calls were made since the last time this method was called.
+     */
+    long minNumberOfWriteBytesRemaining();
+
+    /**
+     * @return the total capacity in bytes
+     */
+    long capacity();
+
+    long getAndClearWriteCount();
+
+    long getAndClearMissedWriteCount();
+
+    long getAndClearContentionCount();
+
+    List<RingBufferReaderStats> readers();
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/BytesStore.java

@@ -0,0 +1,570 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.io.IORuntimeException;
+import net.openhft.chronicle.core.io.ReferenceCounted;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+import javax.crypto.Cipher;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.BufferOverflowException;
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+
+import static java.lang.Math.min;
+
+/**
+ * A immutable reference to some bytes with fixed extents. This can be shared safely across thread
+ * provided the data referenced is accessed in a thread safe manner. Only offset access within the
+ * capacity is possible.
+ */
+@SuppressWarnings({"rawtypes", "unchecked"})
+public interface BytesStore<B extends BytesStore<B, Underlying>, Underlying>
+        extends RandomDataInput, RandomDataOutput<B>, ReferenceCounted, CharSequence {
+
+    /**
+     * This method builds a BytesStore using the bytes in a CharSequence. This chars are encoded
+     * using ISO_8859_1
+     *
+     * @param cs to convert
+     * @return BytesStore
+     */
+    static BytesStore from( CharSequence cs) {
+        if (cs instanceof BytesStore)
+            return ((BytesStore) cs).copy();
+        return HeapBytesStore.wrap(cs.toString().getBytes(StandardCharsets.ISO_8859_1));
+    }
+
+    /**
+     * Wraps a byte[].  This means there is one copy in memory.
+     *
+     * @param bytes to wrap
+     * @return BytesStore
+     */
+    static HeapBytesStore<byte[]> wrap( byte[] bytes) {
+        return HeapBytesStore.wrap(bytes);
+    }
+
+    /**
+     * Wraps a ByteBuffer which can be either on heap or off heap.
+     *
+     * @param bb to wrap
+     * @return BytesStore
+     */
+
+    static BytesStore<?, ByteBuffer> wrap( ByteBuffer bb) {
+        return bb.isDirect()
+                ? NativeBytesStore.wrap(bb)
+                : HeapBytesStore.wrap(bb);
+    }
+
+    /**
+     * @return a PointerBytesStore which can be set to any addressForRead
+     */
+
+    static PointerBytesStore nativePointer() {
+        return new PointerBytesStore();
+    }
+
+    /**
+     * Return the addressForRead and length as a BytesStore
+     *
+     * @param address for the start
+     * @param length  of data
+     * @return as a BytesStore
+     */
+
+    static PointerBytesStore wrap(long address, long length) {
+         PointerBytesStore pbs = nativePointer();
+        pbs.set(address, length);
+        return pbs;
+    }
+
+    /**
+     * @return an empty, fixed sized Bytes
+     */
+    static BytesStore empty() {
+        return NoBytesStore.noBytesStore();
+    }
+
+    /**
+     * @return whether it uses direct memory or not.
+     */
+    @Override
+    boolean isDirectMemory();
+
+    /**
+     * @return a copy of this BytesStore.
+     */
+    BytesStore<B, Underlying> copy();
+
+    /**
+     * @return a Bytes to wrap this ByteStore from the start() to the realCapacity().
+     * @throws IllegalStateException if this Bytes has been released.
+     */
+    @Override
+
+    default Bytes<Underlying> bytesForRead() throws IllegalStateException {
+        try {
+            Bytes<Underlying> ret = bytesForWrite();
+            ret.readLimit(writeLimit());
+            ret.writeLimit(realCapacity());
+            ret.readPosition(start());
+            return ret;
+        } catch (BufferUnderflowException e) {
+            throw new IllegalStateException(e);
+        }
+    }
+
+    /**
+     * @return a Bytes for writing to this BytesStore
+     */
+    @Override
+
+    default Bytes<Underlying> bytesForWrite() throws IllegalStateException {
+        return new VanillaBytes<>(this, writePosition(), writeLimit());
+    }
+
+    /**
+     * Returns if the {@code readPosition} is at the {@code start} and
+     * the {@code writeLimit} is at the {@code end}.
+     * <p>
+     * I.e {@code start() == readPosition() &amp;&amp; writeLimit() == capacity()}
+     *
+     * @return if the {@code readPosition} is at the {@code start} and
+     * the {@code writeLimit} is at the {@code end}
+     */
+    default boolean isClear() {
+        return true;
+    }
+
+    /**
+     * @return the actual capacity available before resizing.
+     */
+    @Override
+    default long realCapacity() {
+        return capacity();
+    }
+
+    /**
+     * @return The maximum limit you can set.
+     */
+    @Override
+    long capacity();
+
+    /**
+     * @return the underlying object being wrapped, if there is one, or null if not.
+     */
+
+    Underlying underlyingObject();
+
+    /**
+     * Use this test to determine if an offset is considered safe.
+     */
+    default boolean inside(long offset) {
+        return start() <= offset && offset < safeLimit();
+    }
+
+    default boolean inside(long offset, long buffer) {
+        return start() <= offset && offset + buffer < safeLimit();
+    }
+
+    /**
+     * @return how many bytes can be safely read, i.e. what is the real capacity of the underlying data.
+     */
+    default long safeLimit() {
+        return capacity();
+    }
+
+    /**
+     * Copy the data to another BytesStore  as long as there is space available in the destination store.
+     *
+     * @param store to copy to
+     * @return how many bytes were copied
+     */
+    default long copyTo( BytesStore store) {
+        long readPos = readPosition();
+        long writePos = store.writePosition();
+        long copy = min(readRemaining(), store.capacity());
+        long i = 0;
+        try {
+            for (; i < copy - 7; i += 8)
+                store.writeLong(writePos + i, readLong(readPos + i));
+            for (; i < copy; i++)
+                store.writeByte(writePos + i, readByte(readPos + i));
+        } catch (BufferOverflowException | BufferUnderflowException e) {
+            throw new AssertionError(e);
+        }
+        return copy;
+    }
+
+    default void copyTo( OutputStream out) throws IOException {
+        BytesInternal.copy(this, out);
+    }
+
+    /**
+     * Fill the BytesStore with zeros
+     *
+     * @param start first byte inclusive
+     * @param end   last byte exclusive.
+     * @return this.
+     */
+    @Override
+
+    default B zeroOut(long start, long end) {
+        if (end <= start)
+            return (B) this;
+        if (start < start())
+            start = start();
+        if (end > capacity())
+            end = capacity();
+        long i = start;
+        try {
+            for (; i < end - 7; i += 8L)
+                writeLong(i, 0L);
+            for (; i < end; i++)
+                writeByte(i, 0);
+        } catch (BufferOverflowException | IllegalArgumentException e) {
+            throw new AssertionError(e);
+        }
+        return (B) this;
+    }
+
+    /**
+     * This method is inherited from CharSequence so result should be the length of the contained
+     * chars sequence although it actually returns the number of underlying bytes. These 2 numbers are only the same
+     * if the encoding we are using is single char for single byte.
+     *
+     * @return length in bytes to read or Integer.MAX_VALUE if longer.
+     */
+    @Override
+    default int length() {
+        return (int) Math.min(Integer.MAX_VALUE, readRemaining());
+    }
+
+    /**
+     * Assume ISO-8859-1 encoding, subclasses can override this.
+     */
+    @Override
+    default char charAt(int index) throws IndexOutOfBoundsException {
+        try {
+            return (char) readUnsignedByte(readPosition() + index);
+
+        } catch (BufferUnderflowException e) {
+            throw new IndexOutOfBoundsException((readPosition() + index) + " >= " + readLimit());
+        }
+    }
+
+    /**
+     * Not supported.
+     */
+
+    @Override
+    default CharSequence subSequence(int start, int end) {
+        throw new UnsupportedOperationException("todo");
+    }
+
+    /**
+     * By default the maximum length of data shown is 256 characters. Use toDebugString(long) if you want more.
+     *
+     * @return This BytesStore as a DebugString.
+     */
+
+    default String toDebugString() {
+        return toDebugString(512);
+    }
+
+    /**
+     * @param maxLength the maximum len of the output
+     * @return This BytesStore as a DebugString.
+     */
+
+    default String toDebugString(long maxLength) {
+        return BytesInternal.toDebugString(this, maxLength);
+    }
+
+    /**
+     * @return the underlying BytesStore
+     */
+
+    default BytesStore bytesStore() {
+        return this;
+    }
+
+    /**
+     * Check if a portion of a BytesStore matches this one.
+     *
+     * @param bytesStore to match against
+     * @param length     to match.
+     * @return true      if the bytes up to min(length, this.length(), bytesStore.length()) matched.
+     */
+    default boolean equalBytes( BytesStore bytesStore, long length)
+            throws BufferUnderflowException {
+        return length == 8 && bytesStore.length() >= 8
+                ? readLong(readPosition()) == bytesStore.readLong(bytesStore.readPosition())
+                : BytesInternal.equalBytesAny(this, bytesStore, length);
+    }
+
+    /**
+     * Return the bytes sum of the readable bytes.
+     *
+     * @return unsigned byte sum.
+     */
+    default int byteCheckSum() throws IORuntimeException {
+        return byteCheckSum(readPosition(), readLimit());
+    }
+
+    default int byteCheckSum(long start, long end) {
+        int sum = 0;
+        for (long i = start; i < end; i++) {
+            sum += readByte(i);
+        }
+        return sum & 0xFF;
+    }
+
+
+    /**
+     * Does the BytesStore end with a character?
+     *
+     * @param c to look for
+     * @return true if its the last character.
+     */
+    default boolean endsWith(char c) {
+        try {
+            return readRemaining() > 0 && readUnsignedByte(readLimit() - 1) == c;
+        } catch (BufferUnderflowException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+    /**
+     * Does the BytesStore start with a character?
+     *
+     * @param c to look for
+     * @return true if its the last character.
+     */
+    default boolean startsWith(char c) {
+        try {
+            return readRemaining() > 0 && readUnsignedByte(readPosition()) == c;
+        } catch (BufferUnderflowException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+    /**
+     * Compare the contents of the BytesStores.
+     *
+     * @param bytesStore to compare with
+     * @return true if they contain the same data.
+     */
+    default boolean contentEquals( BytesStore bytesStore) {
+        return BytesInternal.contentEqual(this, bytesStore);
+    }
+
+    default boolean startsWith( BytesStore bytesStore) {
+        return BytesInternal.startsWith(this, bytesStore);
+    }
+
+
+    default String to8bitString() throws IllegalArgumentException {
+        return BytesInternal.to8bitString(this);
+    }
+
+
+    /**
+     * Perform a <i>not</i> atomic add and get operation for an unsigned byte value. This method
+     * <i>does not</i> check for unsigned byte overflow.
+     *
+     * @param offset to add and get
+     * @param adding value to add, can be 1
+     * @return the sum
+     */
+    default int addAndGetUnsignedByteNotAtomic(long offset, int adding) throws BufferUnderflowException {
+        try {
+            int r = (readUnsignedByte(offset) + adding) & 0xFF;
+            writeByte(offset, (byte) r);
+            return r;
+        } catch (BufferOverflowException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+    /**
+     * Perform a <i>not</i> atomic add and get operation for a short value.
+     *
+     * @param offset to add and get
+     * @param adding value to add, can be 1
+     * @return the sum
+     */
+    default short addAndGetShortNotAtomic(long offset, short adding) throws BufferUnderflowException {
+        try {
+            short r = (short) (readShort(offset) + adding);
+            writeByte(offset, r);
+            return r;
+        } catch (BufferOverflowException | IllegalArgumentException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+    /**
+     * Perform a <i>not</i> atomic add and get operation for an int value.
+     *
+     * @param offset to add and get
+     * @param adding value to add, can be 1
+     * @return the sum
+     */
+    default int addAndGetIntNotAtomic(long offset, int adding) throws BufferUnderflowException {
+        try {
+            int r = readInt(offset) + adding;
+            writeInt(offset, r);
+            return r;
+        } catch (BufferOverflowException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+    /**
+     * Perform a <i>not</i> atomic add and get operation for a float value.
+     *
+     * @param offset to add and get
+     * @param adding value to add, can be 1
+     * @return the sum
+     */
+    default double addAndGetDoubleNotAtomic(long offset, double adding) throws BufferUnderflowException {
+        try {
+            double r = readDouble(offset) + adding;
+            writeDouble(offset, r);
+            return r;
+        } catch (BufferOverflowException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+    /**
+     * Perform a <i>not</i> atomic add and get operation for a float value.
+     *
+     * @param offset to add and get
+     * @param adding value to add, can be 1
+     * @return the sum
+     */
+    default float addAndGetFloatNotAtomic(long offset, float adding) throws BufferUnderflowException {
+        try {
+            float r = readFloat(offset) + adding;
+            writeFloat(offset, r);
+            return r;
+        } catch (BufferOverflowException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+    void move(long from, long to, long length) throws BufferUnderflowException;
+
+    /**
+     * Write a value which is not smaller.
+     *
+     * @param offset  to write to
+     * @param atLeast value it is at least.
+     */
+    default void writeMaxLong(long offset, long atLeast) throws BufferUnderflowException {
+        try {
+            for (; ; ) {
+                long v = readVolatileLong(offset);
+                if (v >= atLeast)
+                    return;
+                if (compareAndSwapLong(offset, v, atLeast))
+                    return;
+            }
+        } catch (BufferOverflowException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+    /**
+     * Write a value which is not smaller.
+     *
+     * @param offset  to write to
+     * @param atLeast value it is at least.
+     */
+    default void writeMaxInt(long offset, int atLeast) throws BufferUnderflowException {
+        try {
+            for (; ; ) {
+                int v = readVolatileInt(offset);
+                if (v >= atLeast)
+                    return;
+                if (compareAndSwapInt(offset, v, atLeast))
+                    return;
+            }
+        } catch (BufferOverflowException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+    default boolean isEmpty() {
+        return readRemaining() == 0;
+    }
+
+    default void cipher( Cipher cipher,  Bytes outBytes,  ByteBuffer using1,  ByteBuffer using2) throws IllegalStateException {
+        long readPos = outBytes.readPosition();
+        try {
+            long writePos = outBytes.writePosition();
+            BytesStore inBytes;
+            long size = readRemaining();
+            if (this.isDirectMemory()) {
+                inBytes = this;
+            } else {
+                inBytes = NativeBytesStore.nativeStore(size);
+                this.copyTo(inBytes);
+            }
+            BytesInternal.assignBytesStoreToByteBuffer(inBytes, using1);
+            int outputSize = cipher.getOutputSize(Math.toIntExact(size));
+            outBytes.ensureCapacity(writePos + outputSize);
+            outBytes.readPositionRemaining(writePos, outputSize);
+            BytesInternal.assignBytesStoreToByteBuffer(outBytes, using2);
+            int len = cipher.update(using1, using2);
+            len += cipher.doFinal(using1, using2);
+            assert len == using2.position();
+            outBytes.writePosition(writePos + using2.position());
+
+        } catch ( Exception e) {
+            throw new IllegalStateException(e);
+        } finally {
+            try {
+                outBytes.readPosition(readPos);
+            } catch (BufferUnderflowException e) {
+                //noinspection ThrowFromFinallyBlock
+                throw new IllegalStateException(e);
+            }
+        }
+    }
+
+    default void cipher( Cipher cipher,  Bytes outBytes) throws IllegalStateException {
+        cipher(cipher, outBytes, BytesInternal.BYTE_BUFFER_TL.get(), BytesInternal.BYTE_BUFFER2_TL.get());
+    }
+
+    /**
+     * @return whether this BytesStore is writable.
+     */
+    default boolean readWrite() {
+        return true;
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/BytesTextMethodTester.java

@@ -0,0 +1,115 @@
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.Jvm;
+import org.jetbrains.annotations.NotNull;
+
+import java.io.IOException;
+import java.util.function.Function;
+
+@SuppressWarnings("rawtypes")
+public class BytesTextMethodTester<T> {
+    private final String input;
+    private final Class<T> outputClass;
+    private final String output;
+    private final Function<T, Object> componentFunction;
+
+    private String setup;
+    private Function<String, String> afterRun;
+
+    private String expected;
+    private String actual;
+
+    public BytesTextMethodTester(String input, Function<T, Object> componentFunction, Class<T> outputClass, String output) {
+        this.input = input;
+        this.outputClass = outputClass;
+        this.output = output;
+        this.componentFunction = componentFunction;
+    }
+
+    public String setup() {
+        return setup;
+    }
+
+
+    public BytesTextMethodTester setup(String setup) {
+        this.setup = setup;
+        return this;
+    }
+
+    public Function<String, String> afterRun() {
+        return afterRun;
+    }
+
+
+    public BytesTextMethodTester afterRun(Function<String, String> afterRun) {
+        this.afterRun = afterRun;
+        return this;
+    }
+
+
+    public BytesTextMethodTester run() throws IOException {
+
+        Bytes<?> bytes2 = new HexDumpBytes();
+        T writer = bytes2.bytesMethodWriter(outputClass);
+
+        Object component = componentFunction.apply(writer);
+        Object[] components = component instanceof Object[]
+                ? (Object[]) component
+                : new Object[]{component};
+
+        if (setup != null) {
+            Bytes bytes0 = HexDumpBytes.fromText(BytesUtil.readFile(setup));
+
+            BytesMethodReader reader0 = bytes0.bytesMethodReaderBuilder()
+                    .defaultParselet(this::unknownMessageId)
+                    .build(components);
+            while (reader0.readOne()) {
+                bytes2.clear();
+            }
+            bytes2.clear();
+        }
+
+        // expected
+        expected = BytesUtil.readFile(output).toString().trim().replace("\r", "");
+
+        Bytes text = BytesUtil.readFile(input);
+        for (String text2 : text.toString().split("###[^\n]*\n")) {
+            if (text2.trim().length() <= 0)
+                continue;
+            Bytes bytes = HexDumpBytes.fromText(text2);
+
+            BytesMethodReader reader = bytes.bytesMethodReaderBuilder()
+                    .defaultParselet(this::unknownMessageId)
+                    .build(components);
+
+            while (reader.readOne()) {
+                if (bytes.readRemaining() > 1)
+                    bytes2.comment("## End Of Message");
+            }
+            bytes.releaseLast();
+            bytes2.comment("## End Of Block");
+        }
+        bytes2.comment("## End Of Test");
+
+        actual = bytes2.toHexString().trim();
+        if (afterRun != null) {
+            expected = afterRun.apply(expected);
+            actual = afterRun.apply(actual);
+        }
+        bytes2.releaseLast();
+        return this;
+    }
+
+    private void unknownMessageId(long id, BytesIn b) {
+        Jvm.warn().on(getClass(), "Unknown message id " + Long.toHexString(id));
+        b.readPosition(b.readLimit());
+    }
+
+    public String expected() {
+        return expected;
+    }
+
+    public String actual() {
+        return actual;
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/BytesUtil.java

@@ -0,0 +1,331 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.ClassLocal;
+import net.openhft.chronicle.core.Jvm;
+import net.openhft.chronicle.core.Maths;
+import net.openhft.chronicle.core.UnsafeMemory;
+import net.openhft.chronicle.core.io.AbstractReferenceCounted;
+import net.openhft.chronicle.core.io.IORuntimeException;
+import net.openhft.chronicle.core.io.IOTools;
+import net.openhft.chronicle.core.util.StringUtils;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+import sun.misc.Unsafe;
+
+import java.io.*;
+import java.lang.reflect.Field;
+import java.lang.reflect.Modifier;
+import java.net.URL;
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+import static net.openhft.chronicle.core.io.IOTools.*;
+
+@SuppressWarnings("rawtypes")
+public enum BytesUtil {
+    ;
+    private static final int[] NO_INTS = {};
+    private static final ClassLocal<int[]> TRIVIALLY_COPYABLE = ClassLocal.withInitial(BytesUtil::isTriviallyCopyable0);
+
+    /**
+     * Is the whole class trivially copyable
+     *
+     * @param clazz to check
+     * @return true if the whole class is trivially copyable
+     */
+    public static boolean isTriviallyCopyable(Class clazz) {
+        return TRIVIALLY_COPYABLE.get(clazz).length > 0;
+    }
+
+    static int[] isTriviallyCopyable0(Class clazz) {
+        if (clazz.isArray()) {
+            Class componentType = clazz.getComponentType();
+            if (componentType.isPrimitive())
+                return new int[]{UnsafeMemory.UNSAFE.arrayBaseOffset(clazz)};
+            return NO_INTS;
+        }
+        List<Field> fields = new ArrayList<>();
+        while (clazz != null && clazz != Object.class) {
+            Collections.addAll(fields, clazz.getDeclaredFields());
+            clazz = clazz.getSuperclass();
+        }
+        int min = Integer.MAX_VALUE;
+        int max = Integer.MIN_VALUE;
+        for (Field field : fields) {
+            int modifiers = field.getModifiers();
+            if (Modifier.isStatic(modifiers))
+                continue;
+            long offset2 = UnsafeMemory.UNSAFE.objectFieldOffset(field);
+            int size = sizeOf(field.getType());
+            min = (int) Math.min(min, offset2);
+            max = (int) Math.max(max, offset2 + size);
+            if (Modifier.isTransient(modifiers))
+                return NO_INTS;
+            if (!field.getType().isPrimitive())
+                return NO_INTS;
+        }
+        return new int[]{min, max};
+    }
+
+    /**
+     * Are all the fields in the range given trivially copyable
+     *
+     * @param clazz to check
+     * @return true if the fields in range are trivially copyable.
+     */
+    public static boolean isTriviallyCopyable(Class clazz, int offset, int length) {
+        int[] ints = TRIVIALLY_COPYABLE.get(clazz);
+        if (ints.length == 0)
+            return false;
+        return offset >= ints[0] && (ints.length == 1 || offset + length <= ints[1]);
+    }
+
+    public static int[] triviallyCopyableRange(Class clazz) {
+        return TRIVIALLY_COPYABLE.get(clazz);
+    }
+
+    private static int sizeOf(Class<?> type) {
+        return type == boolean.class || type == byte.class ? 1
+                : type == short.class || type == char.class ? 2
+                : type == int.class || type == float.class ? 4
+                : type == long.class || type == double.class ? 8
+                : Unsafe.ARRAY_OBJECT_INDEX_SCALE;
+    }
+
+    public static String findFile( String name) throws FileNotFoundException {
+        File file = new File(name);
+        URL url = null;
+        if (!file.exists()) {
+            url = urlFor(name);
+            String file2 = url.getFile()
+                    .replace("target/test-classes", "src/test/resources");
+            file = new File(file2);
+        }
+        if (!file.exists())
+            throw new FileNotFoundException(name);
+        return file.getAbsolutePath();
+    }
+
+    public static Bytes readFile( String name) throws IOException {
+        if (name.startsWith("=")) {
+            return Bytes.from(name.substring(1));
+        }
+        File file = new File(name);
+        URL url = null;
+        if (!file.exists()) {
+            url = urlFor(name);
+            file = new File(url.getFile());
+        }
+        return // name.endsWith(".gz") || !file.exists() || OS.isWindows() ?
+                Bytes.wrapForRead(readAsBytes(url == null ? new FileInputStream(file) : open(url)));
+        //: MappedFile.readOnly(file).acquireBytesForRead(0);
+
+    }
+
+    public static void writeFile(String file, Bytes<byte[]> bytes) throws IOException {
+        try (OutputStream os = new FileOutputStream(file)) {
+            os.write(bytes.underlyingObject());
+        }
+    }
+
+    public static boolean bytesEqual(
+             RandomDataInput a, long offset,
+             RandomDataInput second, long secondOffset, long len)
+            throws BufferUnderflowException {
+        long i = 0;
+        while (len - i >= 8L) {
+            if (a.readLong(offset + i) != second.readLong(secondOffset + i))
+                return false;
+            i += 8L;
+        }
+        if (len - i >= 4L) {
+            if (a.readInt(offset + i) != second.readInt(secondOffset + i))
+                return false;
+            i += 4L;
+        }
+        if (len - i >= 2L) {
+            if (a.readShort(offset + i) != second.readShort(secondOffset + i))
+                return false;
+            i += 2L;
+        }
+        if (i < len)
+            return a.readByte(offset + i) == second.readByte(secondOffset + i);
+        return true;
+    }
+
+    public static boolean bytesEqual( CharSequence cs,  RandomDataInput bs, long offset, int length) {
+        if (cs == null || cs.length() != length)
+            return false;
+        for (int i = 0; i < length; i++) {
+            if (cs.charAt(i) != bs.readUnsignedByte(offset + i))
+                return false;
+        }
+        return true;
+    }
+
+    public static boolean equals(Object o1, Object o2) {
+        if (o1 == o2) return true;
+        if (o1 instanceof CharSequence && o2 instanceof CharSequence)
+            return StringUtils.isEqual((CharSequence) o1, (CharSequence) o2);
+        return o1 != null && o1.equals(o2);
+    }
+
+    public static int asInt( String str) {
+         ByteBuffer bb = ByteBuffer.wrap(str.getBytes(StandardCharsets.ISO_8859_1)).order(ByteOrder.nativeOrder());
+        return bb.getInt();
+    }
+
+    public static int stopBitLength(long n) {
+        if ((n & ~0x7F) == 0) {
+            return 1;
+        }
+        if ((n & ~0x3FFF) == 0) {
+            return 2;
+        }
+        return BytesInternal.stopBitLength0(n);
+    }
+
+
+    public static char[] toCharArray( Bytes bytes) {
+         final char[] chars = new char[Maths.toUInt31(bytes.readRemaining())];
+
+        for (int i = 0; i < bytes.readRemaining(); i++) {
+            chars[i] = (char) bytes.readUnsignedByte(i + bytes.readPosition());
+        }
+        return chars;
+    }
+
+
+    public static char[] toCharArray( Bytes bytes, long position, int length) {
+         final char[] chars = new char[length];
+
+        int j = 0;
+        for (int i = 0; i < length; i++) {
+            chars[j++] = (char) bytes.readUnsignedByte(position + i);
+        }
+        return chars;
+    }
+
+    public static long readStopBit( StreamingDataInput in) throws IORuntimeException {
+        return BytesInternal.readStopBit(in);
+    }
+
+    public static void writeStopBit( StreamingDataOutput out, long n) {
+        BytesInternal.writeStopBit(out, n);
+    }
+
+    public static void parseUtf8(
+             StreamingDataInput in, Appendable appendable, int utflen)
+            throws UTFDataFormatRuntimeException {
+        BytesInternal.parseUtf8(in, appendable, true, utflen);
+    }
+
+    public static void appendUtf8( StreamingDataOutput out,  CharSequence cs) {
+        BytesInternal.appendUtf8(out, cs, 0, cs.length());
+    }
+
+    // used by Chronicle FIX.
+    public static void appendBytesFromStart( Bytes bytes, long startPosition,  StringBuilder sb) {
+        try {
+            BytesInternal.parse8bit(startPosition, bytes, sb, (int) (bytes.readPosition() - startPosition));
+            sb.append('\u2016');
+            sb.append(bytes);
+        } catch (IOException e) {
+            throw new IORuntimeException(e);
+        }
+    }
+
+    public static void readMarshallable( ReadBytesMarshallable marshallable, BytesIn bytes) {
+        BytesMarshaller.BYTES_MARSHALLER_CL.get(marshallable.getClass())
+                .readMarshallable(marshallable, bytes);
+    }
+
+    public static void writeMarshallable( WriteBytesMarshallable marshallable, BytesOut bytes) {
+        BytesMarshaller.BYTES_MARSHALLER_CL.get(marshallable.getClass())
+                .writeMarshallable(marshallable, bytes);
+    }
+
+    @Deprecated(/* to be removed in x.22 */)
+    public static long utf8Length( CharSequence toWrite) {
+        return AppendableUtil.findUtf8Length(toWrite);
+    }
+
+    static String asString(String s, Throwable t) {
+        StringWriter sw = new StringWriter();
+        sw.append(s).append("\n");
+        t.printStackTrace(new PrintWriter(sw));
+        return sw.toString();
+    }
+
+    // calls the BackgroundResourceReleaser and AbstractCloseable.assertCloseableClose first.
+    public static void checkRegisteredBytes() {
+        AbstractReferenceCounted.assertReferencesReleased();
+    }
+
+    public static boolean byteToBoolean(byte b) {
+        return b != 0 && b != 'N' && b != 'n';
+    }
+
+    public static long roundUpTo64ByteAlign(long x) {
+        return (x + 63L) & ~63L;
+    }
+
+    public static long roundUpTo8ByteAlign(long x) {
+        return (x + 7L) & ~7L;
+    }
+
+    public static void read8ByteAlignPadding(Bytes<?> bytes) {
+        bytes.readPosition(roundUpTo8ByteAlign(bytes.readPosition()));
+    }
+
+    public static void write8ByteAlignPadding(Bytes<?> bytes) {
+        long start = bytes.writePosition();
+        long end = roundUpTo8ByteAlign(start);
+        bytes.writePosition(end);
+        bytes.zeroOut(start, end);
+    }
+
+    public static String toDebugString( RandomDataInput bytes, long start, long maxLength) {
+        BytesStore bytes2 = bytes.subBytes(start, maxLength);
+        return bytes2.toDebugString(maxLength);
+    }
+
+    @Deprecated(/* to be removed in x.22 */)
+    public static boolean unregister(BytesStore bs) {
+        IOTools.unmonitor(bs);
+        return true;
+    }
+
+    static class WarnUncheckedElasticBytes {
+        static {
+            Jvm.debug().on(WarnUncheckedElasticBytes.class, "Wrapping elastic bytes with unchecked() will require calling ensureCapacity() as needed!");
+        }
+
+        static void warn() {
+            // static block does the work.
+        }
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/CommonMarshallable.java

@@ -0,0 +1,30 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.annotation.DontChain;
+
+@DontChain
+public interface CommonMarshallable {
+    /**
+     * @return whether this message should be written as self describing
+     */
+    default boolean usesSelfDescribingMessage() {
+        return false;
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/ConnectionDroppedException.java

@@ -0,0 +1,39 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.io.IORuntimeException;
+
+/**
+ * thrown when the TcpChannelHub drops its connection to the server
+ */
+// TODO Move to network where it is used.
+public class ConnectionDroppedException extends IORuntimeException {
+    public ConnectionDroppedException(String message) {
+        super(message);
+    }
+
+    public ConnectionDroppedException(Throwable e) {
+        super(e);
+    }
+
+    public ConnectionDroppedException(String s, Throwable e) {
+        super(s, e);
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/DynamicallySized.java

@@ -0,0 +1,25 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+/**
+ * Marker interface for implementors of {@link Byteable} that indicates
+ * that size requirements can only be calculated on a user-supplied instance.
+ */
+public interface DynamicallySized {
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/GuardedNativeBytes.java

@@ -0,0 +1,179 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.io.IORuntimeException;
+import org.jetbrains.annotations.NotNull;
+
+import java.nio.BufferOverflowException;
+import java.nio.BufferUnderflowException;
+
+import static net.openhft.chronicle.bytes.BinaryWireCode.*;
+
+public class GuardedNativeBytes<Underlying> extends NativeBytes<Underlying> {
+    static final byte BYTE_T = (byte) INT8;
+    static final byte SHORT_T = (byte) INT16;
+    static final byte INT_T = (byte) INT32;
+    static final byte LONG_T = (byte) INT64;
+    static final byte STOP_T = (byte) STOP_BIT;
+    static final byte FLOAT_T = (byte) FLOAT32;
+    static final byte DOUBLE_T = (byte) FLOAT64;
+
+    private static final String[] STRING_FOR_CODE = _stringForCode(GuardedNativeBytes.class);
+
+    public GuardedNativeBytes( BytesStore store, long capacity) throws IllegalStateException {
+        super(store, capacity);
+    }
+
+
+    @Override
+    public Bytes<Underlying> writeByte(byte i8) throws BufferOverflowException {
+        super.writeByte(BYTE_T);
+        return super.writeByte(i8);
+    }
+
+    @Override
+    public Bytes<Underlying> rawWriteByte(byte i8) throws BufferOverflowException {
+        return super.writeByte(i8);
+    }
+
+    @Override
+    public Bytes<Underlying> rawWriteInt(int i) throws BufferOverflowException {
+        return super.writeInt(i);
+    }
+
+    @Override
+    public byte readByte() {
+        expectByte(BYTE_T);
+        return super.readByte();
+    }
+
+    @Override
+    public byte rawReadByte() {
+        return super.readByte();
+    }
+
+    @Override
+    public int rawReadInt() {
+        return super.readInt();
+    }
+
+    @Override
+    public int readUnsignedByte() {
+        expectByte(BYTE_T);
+        return super.readUnsignedByte();
+    }
+
+
+    @Override
+    public Bytes<Underlying> writeShort(short i16) throws BufferOverflowException {
+        super.writeByte(SHORT_T);
+        return super.writeShort(i16);
+    }
+
+    @Override
+    public short readShort() throws BufferUnderflowException {
+        expectByte(SHORT_T);
+        return super.readShort();
+    }
+
+
+    @Override
+    public Bytes<Underlying> writeStopBit(char x) throws BufferOverflowException {
+        super.writeByte(STOP_T);
+        return super.writeStopBit(x);
+    }
+
+
+    @Override
+    public Bytes<Underlying> writeStopBit(long x) throws BufferOverflowException {
+        super.writeByte(STOP_T);
+        return super.writeStopBit(x);
+    }
+
+    @Override
+    public long readStopBit() throws IORuntimeException {
+        expectByte(STOP_T);
+        return super.readStopBit();
+    }
+
+    @Override
+    public char readStopBitChar() throws IORuntimeException {
+        expectByte(STOP_T);
+        return super.readStopBitChar();
+    }
+
+
+    @Override
+    public Bytes<Underlying> writeInt(int i) throws BufferOverflowException {
+        super.writeByte(INT_T);
+        return super.writeInt(i);
+    }
+
+    @Override
+    public int readInt() throws BufferUnderflowException {
+        expectByte(INT_T);
+        return super.readInt();
+    }
+
+
+    @Override
+    public Bytes writeLong(long i64) throws BufferOverflowException {
+        super.writeByte(LONG_T);
+        return super.writeLong(i64);
+    }
+
+    @Override
+    public long readLong() throws BufferUnderflowException {
+        expectByte(LONG_T);
+        return super.readLong();
+    }
+
+
+    @Override
+    public Bytes<Underlying> writeFloat(float f) throws BufferOverflowException {
+        super.writeByte(FLOAT_T);
+        return super.writeFloat(f);
+    }
+
+    @Override
+    public float readFloat() throws BufferUnderflowException {
+        expectByte(FLOAT_T);
+        return super.readFloat();
+    }
+
+
+    @Override
+    public Bytes<Underlying> writeDouble(double d) throws BufferOverflowException {
+        super.writeByte(DOUBLE_T);
+        return super.writeDouble(d);
+    }
+
+    @Override
+    public double readDouble() throws BufferUnderflowException {
+        expectByte(DOUBLE_T);
+        return super.readDouble();
+    }
+
+    private void expectByte(byte expected) {
+        byte type = super.readByte();
+        if (type != expected)
+            throw new IllegalStateException("Expected " + STRING_FOR_CODE[expected & 0xFF]
+                    + " but was " + STRING_FOR_CODE[type & 0xFF]);
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/HeapBytesStore.java

@@ -0,0 +1,402 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.bytes.util.DecoratedBufferOverflowException;
+import net.openhft.chronicle.bytes.util.DecoratedBufferUnderflowException;
+import net.openhft.chronicle.core.Jvm;
+import net.openhft.chronicle.core.Maths;
+import net.openhft.chronicle.core.Memory;
+import net.openhft.chronicle.core.OS;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+import java.nio.BufferOverflowException;
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.util.function.Function;
+
+/**
+ * Wrapper for Heap ByteBuffers and arrays.
+ */
+@SuppressWarnings("restriction")
+public class HeapBytesStore<Underlying>
+        extends AbstractBytesStore<HeapBytesStore<Underlying>, Underlying> {
+
+    private static final Memory MEMORY = OS.memory();
+
+    private final Object realUnderlyingObject;
+    private final int dataOffset;
+    private final long capacity;
+
+    private final Underlying underlyingObject;
+
+    private HeapBytesStore( ByteBuffer byteBuffer) {
+        super(false);
+        //noinspection unchecked
+        this.underlyingObject = (Underlying) byteBuffer;
+        this.realUnderlyingObject = byteBuffer.array();
+        this.dataOffset = Jvm.arrayByteBaseOffset() + byteBuffer.arrayOffset();
+        this.capacity = byteBuffer.capacity();
+    }
+
+    private HeapBytesStore( byte  [] byteArray) {
+        super(false);
+        //noinspection unchecked
+        this.underlyingObject = (Underlying) byteArray;
+        this.realUnderlyingObject = byteArray;
+        this.dataOffset = Jvm.arrayByteBaseOffset();
+        this.capacity = byteArray.length;
+    }
+
+    // Used by Chronicle-Map.
+
+    public static HeapBytesStore<byte[]> wrap( byte[] byteArray) {
+        return new HeapBytesStore<>(byteArray);
+    }
+
+    // Used by Chronicle-Map.
+
+    public static HeapBytesStore<ByteBuffer> wrap( ByteBuffer bb) {
+        return new HeapBytesStore<>(bb);
+    }
+
+    @Override
+    public boolean isDirectMemory() {
+        return false;
+    }
+
+    @Override
+    public void move(long from, long to, long length) {
+        if (from < 0 || to < 0) throw new BufferUnderflowException();
+        //noinspection SuspiciousSystemArraycopy
+        System.arraycopy(realUnderlyingObject, Maths.toUInt31(from), realUnderlyingObject, Maths.toUInt31(to), Maths.toUInt31(length));
+    }
+
+
+    @Override
+    public String toString() {
+        try {
+            return BytesInternal.toString(this);
+
+        } catch (IllegalStateException e) {
+            return e.toString();
+        }
+    }
+
+
+    @Override
+    public BytesStore<HeapBytesStore<Underlying>, Underlying> copy() {
+        throw new UnsupportedOperationException("todo");
+    }
+
+    @Override
+    protected void performRelease() {
+        // nothing to do
+    }
+
+    @Override
+    public long capacity() {
+        return capacity;
+    }
+
+
+    @Override
+    public Underlying underlyingObject() {
+        return underlyingObject;
+    }
+
+    @Override
+    public boolean compareAndSwapInt(long offset, int expected, int value) {
+        return MEMORY.compareAndSwapInt(realUnderlyingObject, dataOffset + offset, expected, value);
+    }
+
+    @Override
+    public void testAndSetInt(long offset, int expected, int value) {
+        MEMORY.testAndSetInt(realUnderlyingObject, dataOffset + offset, expected, value);
+    }
+
+    @Override
+    public boolean compareAndSwapLong(long offset, long expected, long value) {
+        return MEMORY.compareAndSwapLong(
+                realUnderlyingObject, dataOffset + offset, expected, value);
+    }
+
+    @Override
+    public byte readByte(long offset) throws BufferUnderflowException {
+        checkOffset(offset, 1);
+        return MEMORY.readByte(realUnderlyingObject, dataOffset + offset);
+    }
+
+    @Override
+    public short readShort(long offset) throws BufferUnderflowException {
+        checkOffset(offset, 2);
+        return MEMORY.readShort(realUnderlyingObject, dataOffset + offset);
+    }
+
+    @Override
+    public int readInt(long offset) throws BufferUnderflowException {
+        checkOffset(offset, 4);
+        return MEMORY.readInt(realUnderlyingObject, dataOffset + offset);
+    }
+
+    @Override
+    public long readLong(long offset) throws BufferUnderflowException {
+        checkOffset(offset, 8);
+        return MEMORY.readLong(realUnderlyingObject, dataOffset + offset);
+    }
+
+    @Override
+    public float readFloat(long offset) throws BufferUnderflowException {
+        checkOffset(offset, 4);
+        return MEMORY.readFloat(realUnderlyingObject, dataOffset + offset);
+    }
+
+    @Override
+    public double readDouble(long offset) throws BufferUnderflowException {
+        checkOffset(offset, 8);
+        return MEMORY.readDouble(realUnderlyingObject, dataOffset + offset);
+    }
+
+    @Override
+    public byte readVolatileByte(long offset) throws BufferUnderflowException {
+        checkOffset(offset, 1);
+        return MEMORY.readVolatileByte(realUnderlyingObject, dataOffset + offset);
+    }
+
+    @Override
+    public short readVolatileShort(long offset) throws BufferUnderflowException {
+        checkOffset(offset, 2);
+        return MEMORY.readVolatileShort(realUnderlyingObject, dataOffset + offset);
+    }
+
+    @Override
+    public int readVolatileInt(long offset) throws BufferUnderflowException {
+        checkOffset(offset, 4);
+        return MEMORY.readVolatileInt(realUnderlyingObject, dataOffset + offset);
+    }
+
+    @Override
+    public long readVolatileLong(long offset) throws BufferUnderflowException {
+        checkOffset(offset, 8);
+        return MEMORY.readVolatileLong(realUnderlyingObject, dataOffset + offset);
+    }
+
+
+    @Override
+    public HeapBytesStore<Underlying> writeByte(long offset, byte b)
+            throws BufferOverflowException {
+        writeCheckOffset(offset, 1);
+        MEMORY.writeByte(realUnderlyingObject, dataOffset + offset, b);
+        return this;
+    }
+
+
+    @Override
+    public HeapBytesStore<Underlying> writeShort(long offset, short i16)
+            throws BufferOverflowException {
+        writeCheckOffset(offset, 2);
+        MEMORY.writeShort(realUnderlyingObject, dataOffset + offset, i16);
+        return this;
+    }
+
+
+    @Override
+    public HeapBytesStore<Underlying> writeInt(long offset, int i32)
+            throws BufferOverflowException {
+        writeCheckOffset(offset, 4);
+        MEMORY.writeInt(realUnderlyingObject, dataOffset + offset, i32);
+        return this;
+    }
+
+
+    @Override
+    public HeapBytesStore<Underlying> writeOrderedInt(long offset, int i32)
+            throws BufferOverflowException {
+        writeCheckOffset(offset, 4);
+        MEMORY.writeOrderedInt(realUnderlyingObject, dataOffset + offset, i32);
+        return this;
+    }
+
+
+    @Override
+    public HeapBytesStore<Underlying> writeLong(long offset, long i64)
+            throws BufferOverflowException {
+        writeCheckOffset(offset, 8);
+        MEMORY.writeLong(realUnderlyingObject, dataOffset + offset, i64);
+        return this;
+    }
+
+
+    @Override
+    public HeapBytesStore<Underlying> writeOrderedLong(long offset, long i)
+            throws BufferOverflowException {
+        writeCheckOffset(offset, 8);
+        MEMORY.writeOrderedLong(realUnderlyingObject, dataOffset + offset, i);
+        return this;
+    }
+
+
+    @Override
+    public HeapBytesStore<Underlying> writeFloat(long offset, float f)
+            throws BufferOverflowException {
+        writeCheckOffset(offset, 4);
+        MEMORY.writeFloat(realUnderlyingObject, dataOffset + offset, f);
+        return this;
+    }
+
+
+    @Override
+    public HeapBytesStore<Underlying> writeDouble(long offset, double d)
+            throws BufferOverflowException {
+        writeCheckOffset(offset, 8);
+        MEMORY.writeDouble(realUnderlyingObject, dataOffset + offset, d);
+        return this;
+    }
+
+
+    @Override
+    public HeapBytesStore<Underlying> writeVolatileByte(long offset, byte i8)
+            throws BufferOverflowException {
+        writeCheckOffset(offset, 1);
+        MEMORY.writeVolatileByte(realUnderlyingObject, dataOffset + offset, i8);
+        return this;
+    }
+
+
+    @Override
+    public HeapBytesStore<Underlying> writeVolatileShort(long offset, short i16)
+            throws BufferOverflowException {
+        writeCheckOffset(offset, 2);
+        MEMORY.writeVolatileShort(realUnderlyingObject, dataOffset + offset, i16);
+        return this;
+    }
+
+
+    @Override
+    public HeapBytesStore<Underlying> writeVolatileInt(long offset, int i32)
+            throws BufferOverflowException {
+        writeCheckOffset(offset, 4);
+        MEMORY.writeVolatileInt(realUnderlyingObject, dataOffset + offset, i32);
+        return this;
+    }
+
+
+    @Override
+    public HeapBytesStore<Underlying> writeVolatileLong(long offset, long i64)
+            throws BufferOverflowException {
+        writeCheckOffset(offset, 8);
+        MEMORY.writeVolatileLong(realUnderlyingObject, dataOffset + offset, i64);
+        return this;
+    }
+
+
+    @Override
+    public HeapBytesStore<Underlying> write(
+            long offsetInRDO, byte[] bytes, int offset, int length) throws BufferOverflowException {
+        writeCheckOffset(offsetInRDO, length);
+        MEMORY.copyMemory(
+                bytes, offset, realUnderlyingObject, this.dataOffset + offsetInRDO, length);
+        return this;
+    }
+
+    @Override
+    public void write(
+            long offsetInRDO,  ByteBuffer bytes, int offset, int length) throws BufferOverflowException {
+        writeCheckOffset(offsetInRDO, length);
+        assert realUnderlyingObject == null || dataOffset >= (Jvm.is64bit() ? 12 : 8);
+        if (bytes.isDirect()) {
+            MEMORY.copyMemory(Jvm.address(bytes), realUnderlyingObject,
+                    this.dataOffset + offsetInRDO, length);
+
+        } else {
+            MEMORY.copyMemory(bytes.array(), offset, realUnderlyingObject,
+                    this.dataOffset + offsetInRDO, length);
+        }
+    }
+
+
+    @Override
+    public HeapBytesStore<Underlying> write(long writeOffset,
+                                             RandomDataInput bytes, long readOffset, long length) {
+        long i;
+        for (i = 0; i < length - 7; i += 8) {
+            long x = bytes.readLong(readOffset + i);
+            writeLong(writeOffset + i, x);
+        }
+        for (; i < length; i++) {
+            byte x = bytes.readByte(readOffset + i);
+            writeByte(writeOffset + i, x);
+        }
+        return this;
+    }
+
+    @Override
+    public long addressForRead(long offset) throws UnsupportedOperationException {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public long addressForWrite(long offset) throws UnsupportedOperationException {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public long addressForWritePosition() throws UnsupportedOperationException, BufferOverflowException {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public void nativeRead(long position, long address, long size) {
+        throw new UnsupportedOperationException("todo");
+    }
+
+    @Override
+    public void nativeWrite(long address, long position, long size) {
+        throw new UnsupportedOperationException("todo");
+    }
+
+    @SuppressWarnings("rawtypes")
+    @Override
+    public boolean equals(Object obj) {
+        return obj instanceof BytesStore && BytesInternal.contentEqual(this, (BytesStore) obj);
+    }
+
+    @Override
+    public boolean sharedMemory() {
+        return false;
+    }
+
+    private void checkOffset(long offset, int size) throws BufferUnderflowException {
+        checkBounds(offset, size, DecoratedBufferUnderflowException::new);
+    }
+
+    private void writeCheckOffset(long offset, int size) throws BufferOverflowException {
+        checkBounds(offset, size, DecoratedBufferOverflowException::new);
+    }
+
+    private void checkBounds(final long offset, final int size,
+                             final Function<String, RuntimeException> exceptionFunction) {
+        if (offset < start() || offset + size > capacity) {
+            throw exceptionFunction.apply(
+                    String.format("Offset: %d, start: %d, size: %d, capacity: %d",
+                            offset, start(), size, capacity));
+        }
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/Invocation.java

@@ -0,0 +1,9 @@
+package net.openhft.chronicle.bytes;
+
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+
+@FunctionalInterface
+public interface Invocation {
+    Object invoke(Method m, Object o, Object[] args) throws InvocationTargetException;
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/MappedBytesStore.java

@@ -0,0 +1,115 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.OS;
+import net.openhft.chronicle.core.io.ReferenceOwner;
+import org.jetbrains.annotations.NotNull;
+
+import java.nio.BufferUnderflowException;
+
+/**
+ * BytesStore to wrap memory mapped data.
+ */
+public class MappedBytesStore extends NativeBytesStore<Void> {
+    private final MappedFile mappedFile;
+    private final long start;
+    private final long safeLimit;
+
+    protected MappedBytesStore(ReferenceOwner owner, MappedFile mappedFile, long start, long address, long capacity, long safeCapacity) throws IllegalStateException {
+        super(address, start + capacity, new OS.Unmapper(address, capacity), false);
+        this.mappedFile = mappedFile;
+        this.start = start;
+        this.safeLimit = start + safeCapacity;
+        reserveTransfer(INIT, owner);
+    }
+
+    /**
+     * Fetch the capacity of the underlying file
+     * This can differ from the exposed capacity() of this bytes store (which has been page aligned)
+     *
+     * @return - capacity of the underlying file
+     */
+    public long underlyingCapacity() {
+        return mappedFile.capacity();
+    }
+
+    @Override
+    public  Bytes<Void> bytesForRead() throws IllegalStateException {
+        try {
+            return new VanillaBytes<Void>(this)
+                    .readLimit(writeLimit())
+                    .readPosition(start());
+        } catch (BufferUnderflowException e) {
+            throw new IllegalStateException(e);
+        }
+    }
+
+
+    @Override
+    public VanillaBytes<Void> bytesForWrite() throws IllegalStateException {
+        return new VanillaBytes<>(this);
+    }
+
+    @Override
+    public boolean inside(long offset) {
+        return start <= offset && offset < safeLimit;
+    }
+
+    @Override
+    public boolean inside(long offset, long buffer) {
+        // this is correct that it uses the maximumLimit, yes it is different than the method above.
+        return start <= offset && offset + buffer < limit;
+    }
+
+    @Override
+    public long safeLimit() {
+        return safeLimit;
+    }
+
+    @Override
+    public byte readByte(long offset) {
+        return memory.readByte(address - start + offset);
+    }
+
+
+    @Override
+    public NativeBytesStore<Void> writeOrderedInt(long offset, int i) {
+        memory.writeOrderedInt(address - start + offset, i);
+        return this;
+    }
+
+    @Override
+    protected long translate(long offset) {
+        assert offset >= start;
+        assert offset < limit;
+
+        return offset - start;
+    }
+
+    @Override
+    public long start() {
+        return start;
+    }
+
+    @Override
+    public long readPosition() {
+        return start();
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/MappedBytesStoreFactory.java

@@ -0,0 +1,28 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.io.ReferenceOwner;
+import org.jetbrains.annotations.NotNull;
+
+@FunctionalInterface
+public interface MappedBytesStoreFactory {
+
+    MappedBytesStore create(ReferenceOwner owner, MappedFile mappedFile, long start, long address, long capacity, long safeCapacity);
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/MappedFile.java

@@ -0,0 +1,541 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.CleaningRandomAccessFile;
+import net.openhft.chronicle.core.Jvm;
+import net.openhft.chronicle.core.OS;
+import net.openhft.chronicle.core.io.AbstractCloseableReferenceCounted;
+import net.openhft.chronicle.core.io.IORuntimeException;
+import net.openhft.chronicle.core.io.ReferenceOwner;
+import net.openhft.chronicle.core.onoes.Slf4jExceptionHandler;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.channels.ClosedByInterruptException;
+import java.nio.channels.FileChannel;
+import java.nio.channels.FileChannel.MapMode;
+import java.nio.channels.FileLock;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.List;
+
+import static net.openhft.chronicle.core.io.Closeable.closeQuietly;
+
+/**
+ * A memory mapped files which can be randomly accessed in chunks. It has overlapping regions to
+ * avoid wasting bytes at the end of chunks.
+ */
+@SuppressWarnings({"rawtypes", "unchecked", "restriction"})
+public class MappedFile extends AbstractCloseableReferenceCounted {
+    static final boolean RETAIN = Jvm.getBoolean("mappedFile.retain");
+    private static final long DEFAULT_CAPACITY = 128L << 40;
+
+
+    private final RandomAccessFile raf;
+    private final FileChannel fileChannel;
+    private final long chunkSize;
+    private final long overlapSize;
+    private final List<MappedBytesStore> stores = new ArrayList<>();
+    private final long capacity;
+
+    private final File file;
+    private final String canonicalPath;
+    private final boolean readOnly;
+    private NewChunkListener newChunkListener = MappedFile::logNewChunk;
+
+    protected MappedFile( final File file,
+                          final RandomAccessFile raf,
+                         final long chunkSize,
+                         final long overlapSize,
+                         final long capacity,
+                         final boolean readOnly) {
+        this.file = file;
+        try {
+            this.canonicalPath = file.getCanonicalPath().intern();
+        } catch (IOException ioe) {
+            throw new IllegalStateException("Unable to obtain the canonical path for " + file.getAbsolutePath(), ioe);
+        }
+        this.raf = raf;
+        this.fileChannel = raf.getChannel();
+        this.chunkSize = OS.mapAlign(chunkSize);
+        this.overlapSize = overlapSize > 0 && overlapSize < 64 << 10 ? chunkSize : OS.mapAlign(overlapSize);
+        this.capacity = capacity;
+        this.readOnly = readOnly;
+
+        Jvm.doNotCloseOnInterrupt(getClass(), this.fileChannel);
+    }
+
+    private static void logNewChunk(final String filename,
+                                    final int chunk,
+                                    final long delayMicros) {
+        if (delayMicros < 100 || !Jvm.isDebugEnabled(MappedFile.class))
+            return;
+
+        // avoid a GC while trying to memory map.
+        final String message = BytesInternal.acquireStringBuilder()
+                .append("Allocation of ").append(chunk)
+                .append(" chunk in ").append(filename)
+                .append(" took ").append(delayMicros / 1e3).append(" ms.")
+                .toString();
+        Jvm.perf().on(MappedFile.class, message);
+    }
+
+
+    public static MappedFile of( final File file,
+                                final long chunkSize,
+                                final long overlapSize,
+                                final boolean readOnly) throws FileNotFoundException {
+//        if (readOnly && OS.isWindows()) {
+//            Jvm.warn().on(MappedFile.class, "Read only mode not supported on Windows, defaulting to read/write");
+//            readOnly = false;
+//        }
+
+         RandomAccessFile raf = new CleaningRandomAccessFile(file, readOnly ? "r" : "rw");
+//        try {
+        final long capacity = /*readOnly ? raf.length() : */DEFAULT_CAPACITY;
+        return new MappedFile(file, raf, chunkSize, overlapSize, capacity, readOnly);
+/*
+        } catch (IOException e) {
+            Closeable.closeQuietly(raf);
+             FileNotFoundException fnfe = new FileNotFoundException("Unable to open " + file);
+            fnfe.initCause(e);
+            throw fnfe;
+        }
+*/
+    }
+
+
+    public static MappedFile mappedFile( final File file, final long chunkSize) throws FileNotFoundException {
+        return mappedFile(file, chunkSize, OS.pageSize());
+    }
+
+/*
+    private void check(Throwable throwable, int[] count) {
+        for (int i = 0; i < stores.size(); i++) {
+            WeakReference<MappedBytesStore> storeRef = stores.get(i);
+            if (storeRef == null)
+                continue;
+             MappedBytesStore mbs = storeRef.get();
+            if (mbs != null && mbs.refCount() > 0) {
+                mbs.releaseLast();
+                throwable.printStackTrace();
+                count[0]++;
+            }
+        }
+    }
+*/
+
+
+    public static MappedFile mappedFile( final String filename, final long chunkSize) throws FileNotFoundException {
+        return mappedFile(filename, chunkSize, OS.pageSize());
+    }
+
+
+    public static MappedFile mappedFile( final String filename,
+                                        final long chunkSize,
+                                        final long overlapSize) throws FileNotFoundException {
+        return mappedFile(new File(filename), chunkSize, overlapSize);
+    }
+
+
+    public static MappedFile mappedFile( final File file,
+                                        final long chunkSize,
+                                        final long overlapSize) throws FileNotFoundException {
+        return mappedFile(file, chunkSize, overlapSize, false);
+    }
+
+
+    public static MappedFile mappedFile( final File file,
+                                        final long chunkSize,
+                                        final long overlapSize,
+                                        final boolean readOnly) throws FileNotFoundException {
+        return MappedFile.of(file, chunkSize, overlapSize, readOnly);
+    }
+
+
+    public static MappedFile readOnly( final File file) throws FileNotFoundException {
+        long chunkSize = file.length();
+        long overlapSize = 0;
+        // Chunks of 4 GB+ not supported on Windows.
+        if (OS.isWindows() && chunkSize > 2L << 30) {
+            chunkSize = 2L << 30;
+            overlapSize = OS.pageSize();
+        }
+        return MappedFile.of(file, chunkSize, overlapSize, true);
+    }
+
+
+    public static MappedFile mappedFile( final File file,
+                                        final long capacity,
+                                        final long chunkSize,
+                                        final long overlapSize,
+                                        final boolean readOnly) throws IOException {
+        final RandomAccessFile raf = new CleaningRandomAccessFile(file, readOnly ? "r" : "rw");
+        // Windows throws an exception when setting the length when you re-open
+        if (raf.length() < capacity)
+            raf.setLength(capacity);
+        return new MappedFile(file, raf, chunkSize, overlapSize, capacity, readOnly);
+    }
+
+    public static void warmup() {
+        final List<IOException> errorsDuringWarmup = new ArrayList<>();
+        try {
+            Jvm.setExceptionHandlers(Slf4jExceptionHandler.FATAL, null, null);
+
+             final File file = File.createTempFile("delete_warming_up", "me");
+            file.deleteOnExit();
+            final long mapAlignment = OS.mapAlignment();
+            final int chunks = 64;
+            final int compileThreshold = Jvm.compileThreshold();
+            for (int j = 0; j <= compileThreshold; j += chunks) {
+                try {
+                    try ( RandomAccessFile raf = new CleaningRandomAccessFile(file, "rw")) {
+                        try (final MappedFile mappedFile = new MappedFile(file, raf, mapAlignment, 0, mapAlignment * chunks, false)) {
+                            warmup0(mapAlignment, chunks, mappedFile);
+                        }
+                    }
+                    Thread.yield();
+                } catch (IOException e) {
+                    errorsDuringWarmup.add(e);
+                }
+            }
+            Thread.yield();
+            Files.delete(file.toPath());
+        } catch (IOException e) {
+            Jvm.resetExceptionHandlers();
+            Jvm.warn().on(MappedFile.class, "Error during warmup", e);
+        } finally {
+            Jvm.resetExceptionHandlers();
+            if (errorsDuringWarmup.size() > 0)
+                Jvm.warn().on(MappedFile.class, errorsDuringWarmup.size() + " errors during warmup: " + errorsDuringWarmup);
+        }
+    }
+
+    private static void warmup0(final long mapAlignment,
+                                final int chunks,
+                                 final MappedFile mappedFile) throws IOException {
+        ReferenceOwner warmup = ReferenceOwner.temporary("warmup");
+        for (int i = 0; i < chunks; i++) {
+            mappedFile.acquireBytesForRead(warmup, i * mapAlignment)
+                    .release(warmup);
+            mappedFile.acquireBytesForWrite(warmup, i * mapAlignment)
+                    .release(warmup);
+        }
+    }
+
+
+    public File file() {
+        return file;
+    }
+
+    /**
+     * @throws IllegalStateException if closed.
+     */
+
+    public MappedBytesStore acquireByteStore(
+            ReferenceOwner owner,
+            final long position)
+            throws IOException, IllegalArgumentException, IllegalStateException {
+        return acquireByteStore(owner, position, null, readOnly ? ReadOnlyMappedBytesStore::new : MappedBytesStore::new);
+    }
+
+
+    public MappedBytesStore acquireByteStore(
+            ReferenceOwner owner,
+            final long position,
+            BytesStore oldByteStore)
+            throws IOException, IllegalArgumentException, IllegalStateException {
+        return acquireByteStore(owner, position, oldByteStore, readOnly ? ReadOnlyMappedBytesStore::new : MappedBytesStore::new);
+    }
+
+
+    public MappedBytesStore acquireByteStore(
+            ReferenceOwner owner,
+            final long position,
+            BytesStore oldByteStore,
+             final MappedBytesStoreFactory mappedBytesStoreFactory)
+            throws IOException,
+            IllegalArgumentException,
+            IllegalStateException {
+
+        throwExceptionIfClosed();
+
+        if (position < 0)
+            throw new IOException("Attempt to access a negative position: " + position);
+        final int chunk = (int) (position / chunkSize);
+        Jvm.safepoint();
+
+        final MappedBytesStore mbs;
+        synchronized (stores) {
+            while (stores.size() <= chunk)
+                stores.add(null);
+            mbs = stores.get(chunk);
+        }
+        if (mbs != null) {
+            // don't reserve it again if we are already holding it.
+            if (mbs == oldByteStore) {
+                return mbs;
+            }
+            if (mbs.tryReserve(owner)) {
+                return mbs;
+            }
+        }
+
+        // its important we perform this outside the synchronized below, as this operation can take a while and if synchronized can block slow tailer
+        // from acquiring the next block
+        resizeRafIfTooSmall(chunk);
+
+        synchronized (stores) {
+
+            // We are back, protected by synchronized, and need to
+            // update our view on previous existence (we might have been stalled
+            // for a long time since we last checked dues to resizing and another
+            // thread might have added a MappedByteStore (very unlikely but still possible))
+            final MappedBytesStore mbs1 = stores.get(chunk);
+            if (mbs1 != null && mbs1.tryReserve(owner)) {
+                return mbs1;
+            }
+            // *** THIS CAN TAKE A LONG TIME IF A RESIZE HAS TO OCCUR ***
+            // let double check it to make sure no other thread change it in the meantime.
+            //resizeRafIfTooSmall(chunk);
+
+            final long mappedSize = chunkSize + overlapSize;
+            final MapMode mode = readOnly ? MapMode.READ_ONLY : MapMode.READ_WRITE;
+            final long startOfMap = chunk * chunkSize;
+
+            final long beginNs = System.nanoTime();
+
+            final long address = OS.map(fileChannel, mode, startOfMap, mappedSize);
+            final MappedBytesStore mbs2 = mappedBytesStoreFactory.create(owner, this, chunk * this.chunkSize, address, mappedSize, this.chunkSize);
+            if (RETAIN)
+                mbs2.reserve(this);
+            stores.set(chunk, mbs2);
+
+            final long elapsedNs = System.nanoTime() - beginNs;
+            if (newChunkListener != null)
+                newChunkListener.onNewChunk(file.getPath(), chunk, elapsedNs / 1000);
+
+            if (elapsedNs >= 2_000_000L)
+                Jvm.perf().on(getClass(), "Took " + elapsedNs / 1_000_000L + " ms to add mapping for " + file());
+
+            return mbs2;
+        }
+    }
+
+    private void resizeRafIfTooSmall(final int chunk) throws IOException {
+        Jvm.safepoint();
+
+        final long minSize = (chunk + 1L) * chunkSize + overlapSize;
+        long size = fileChannel.size();
+        Jvm.safepoint();
+        if (size >= minSize || readOnly)
+            return;
+
+        // handle a possible race condition between processes.
+        try {
+            // A single JVM cannot lock a distinct canonical file more than once.
+
+            // We might have several MappedFile objects that maps to
+            // the same underlying file (possibly via hard or soft links)
+            // so we use the canonical path as a lock key
+
+            // Ensure exclusivity for any and all MappedFile objects handling
+            // the same canonical file.
+            synchronized (canonicalPath) {
+                size = fileChannel.size();
+                if (size < minSize) {
+                    final long beginNs = System.nanoTime();
+                    try (FileLock ignore = fileChannel.lock()) {
+                        size = fileChannel.size();
+                        if (size < minSize) {
+                            Jvm.safepoint();
+                            raf.setLength(minSize);
+                            Jvm.safepoint();
+                        }
+                    }
+                    final long elapsedNs = System.nanoTime() - beginNs;
+                    if (elapsedNs >= 1_000_000L) {
+                        Jvm.perf().on(getClass(), "Took " + elapsedNs / 1000L + " us to grow file " + file());
+                    }
+                }
+            }
+        } catch (IOException ioe) {
+            throw new IOException("Failed to resize to " + minSize, ioe);
+        }
+    }
+
+    /**
+     * Convenience method so you don't need to release the BytesStore
+     */
+
+    public Bytes acquireBytesForRead(ReferenceOwner owner, final long position)
+            throws IOException, IllegalStateException, IllegalArgumentException {
+        throwExceptionIfClosed();
+
+         final MappedBytesStore mbs = acquireByteStore(owner, position, null);
+        final Bytes bytes = mbs.bytesForRead();
+        bytes.readPositionUnlimited(position);
+        bytes.reserveTransfer(INIT, owner);
+        mbs.release(owner);
+        return bytes;
+    }
+
+    public void acquireBytesForRead(ReferenceOwner owner, final long position,  final VanillaBytes bytes)
+            throws IOException, IllegalStateException, IllegalArgumentException {
+        throwExceptionIfClosed();
+
+         final MappedBytesStore mbs = acquireByteStore(owner, position, null);
+        bytes.bytesStore(mbs, position, mbs.capacity() - position);
+    }
+
+
+    public Bytes acquireBytesForWrite(ReferenceOwner owner, final long position)
+            throws IOException, IllegalStateException, IllegalArgumentException {
+        throwExceptionIfClosed();
+
+         MappedBytesStore mbs = acquireByteStore(owner, position, null);
+         Bytes bytes = mbs.bytesForWrite();
+        bytes.writePosition(position);
+        bytes.reserveTransfer(INIT, owner);
+        mbs.release(owner);
+        return bytes;
+    }
+
+    public void acquireBytesForWrite(ReferenceOwner owner, final long position,  final VanillaBytes bytes)
+            throws IOException, IllegalStateException, IllegalArgumentException {
+        throwExceptionIfClosed();
+
+         final MappedBytesStore mbs = acquireByteStore(owner, position, null);
+        bytes.bytesStore(mbs, position, mbs.capacity() - position);
+        bytes.writePosition(position);
+    }
+
+    @Override
+    protected boolean performReleaseInBackground() {
+        // don't perform the close in the background as that just sets a flag. This does the real work.
+        return true;
+    }
+
+    protected void performRelease() {
+        try {
+            synchronized (stores) {
+                for (int i = 0; i < stores.size(); i++) {
+                    final MappedBytesStore mbs = stores.get(i);
+                    if (mbs != null && RETAIN) {
+                        // this MappedFile is the only referrer to the MappedBytesStore at this point,
+                        // so ensure that it is released
+                        mbs.release(this);
+                    }
+                    // Dereference released entities
+                    stores.set(i, null);
+                }
+            }
+        } finally {
+            closeQuietly(raf);
+            setClosed();
+        }
+    }
+
+
+    public String referenceCounts() {
+         final StringBuilder sb = new StringBuilder();
+        sb.append("refCount: ").append(refCount());
+        for ( final MappedBytesStore mbs : stores) {
+            long count = 0;
+            if (mbs != null)
+                count = mbs.refCount();
+            sb.append(", ").append(count);
+        }
+        return sb.toString();
+    }
+
+    public long capacity() {
+        return capacity;
+    }
+
+    public long chunkSize() {
+        return chunkSize;
+    }
+
+    public long overlapSize() {
+        return overlapSize;
+    }
+
+    public NewChunkListener getNewChunkListener() {
+        return newChunkListener;
+    }
+
+    public void setNewChunkListener(final NewChunkListener listener) {
+        throwExceptionIfClosedInSetter();
+
+        this.newChunkListener = listener;
+    }
+
+    public long actualSize() throws IORuntimeException {
+
+        boolean interrupted = Thread.interrupted();
+        try {
+            return fileChannel.size();
+
+            // this was seen once deep in the JVM.
+        } catch (ArrayIndexOutOfBoundsException aiooe) {
+            // try again.
+            return actualSize();
+
+        } catch (ClosedByInterruptException cbie) {
+            close();
+            interrupted = true;
+            throw new IllegalStateException(cbie);
+
+        } catch (IOException e) {
+            final boolean open = fileChannel.isOpen();
+            if (open) {
+                throw new IORuntimeException(e);
+            } else {
+                close();
+                throw new IllegalStateException(e);
+            }
+        } finally {
+            if (interrupted)
+                Thread.currentThread().interrupt();
+        }
+    }
+
+
+    public RandomAccessFile raf() {
+        return raf;
+    }
+
+    @Override
+    protected void finalize() throws Throwable {
+        warnAndReleaseIfNotReleased();
+        super.finalize();
+    }
+
+    @Override
+    protected boolean threadSafetyCheck(boolean isUsed) {
+        // component is thread safe
+        return true;
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/MappedUniqueMicroTimeProvider.java

@@ -0,0 +1,99 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.Jvm;
+import net.openhft.chronicle.core.OS;
+import net.openhft.chronicle.core.io.IORuntimeException;
+import net.openhft.chronicle.core.io.IOTools;
+import net.openhft.chronicle.core.io.ReferenceOwner;
+import net.openhft.chronicle.core.time.SystemTimeProvider;
+import net.openhft.chronicle.core.time.TimeProvider;
+
+import java.io.IOException;
+
+/**
+ * Timestamps are unique across threads/processes on a single machine.
+ *
+ * @deprecated Use {@link MappedUniqueTimeProvider} instead.
+ */
+@Deprecated(/* to be removed in x.23 */)
+public enum MappedUniqueMicroTimeProvider implements TimeProvider {
+    INSTANCE;
+
+    private static final int LAST_TIME = 128;
+
+    private final MappedFile file;
+    @SuppressWarnings("rawtypes")
+    private final Bytes bytes;
+    private TimeProvider provider = SystemTimeProvider.INSTANCE;
+
+    MappedUniqueMicroTimeProvider() {
+        try {
+            String user = System.getProperty("user.name", "unknown");
+            file = MappedFile.mappedFile(OS.TMP + "/.time-stamp." + user + ".dat", OS.pageSize(), 0);
+            IOTools.unmonitor(file);
+            ReferenceOwner mumtp = ReferenceOwner.temporary("mumtp");
+            bytes = file.acquireBytesForWrite(mumtp, 0);
+            bytes.append8bit("&TSF\nTime stamp file uses for sharing a unique id\n");
+            IOTools.unmonitor(bytes);
+        } catch (IOException ioe) {
+            throw new IORuntimeException(ioe);
+        }
+    }
+
+    public MappedUniqueMicroTimeProvider provider(TimeProvider provider) {
+        this.provider = provider;
+        return this;
+    }
+
+    @Override
+    public long currentTimeMillis() {
+        return provider.currentTimeMillis();
+    }
+
+    @Override
+    public long currentTimeMicros() {
+        long timeus = provider.currentTimeMicros();
+        while (true) {
+            long time0 = bytes.readVolatileLong(LAST_TIME);
+            long time0us = time0 / 1000;
+            long time;
+            if (time0us >= timeus)
+                time = (time0us + 1) * 1000;
+            else
+                time = timeus * 1000;
+            if (bytes.compareAndSwapLong(LAST_TIME, time0, time))
+                return time / 1_000;
+            Jvm.nanoPause();
+        }
+    }
+
+    @Override
+    public long currentTimeNanos() {
+        long time = provider.currentTimeNanos();
+        while (true) {
+            long time0 = bytes.readVolatileLong(LAST_TIME);
+            if (time0 >= time)
+                time = time0 + 1;
+            if (bytes.compareAndSwapLong(LAST_TIME, time0, time))
+                return time;
+            Jvm.nanoPause();
+        }
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/MappedUniqueTimeProvider.java

@@ -0,0 +1,104 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.Jvm;
+import net.openhft.chronicle.core.OS;
+import net.openhft.chronicle.core.io.IORuntimeException;
+import net.openhft.chronicle.core.io.IOTools;
+import net.openhft.chronicle.core.io.ReferenceOwner;
+import net.openhft.chronicle.core.time.SystemTimeProvider;
+import net.openhft.chronicle.core.time.TimeProvider;
+
+import java.io.IOException;
+
+/**
+ * Timestamps are unique across threads/processes on a single machine.
+ */
+public enum MappedUniqueTimeProvider implements TimeProvider {
+    INSTANCE;
+
+    private static final int LAST_TIME = 128;
+
+    private final MappedFile file;
+    @SuppressWarnings("rawtypes")
+    private final Bytes bytes;
+    private TimeProvider provider = SystemTimeProvider.INSTANCE;
+
+    MappedUniqueTimeProvider() {
+        try {
+            String user = System.getProperty("user.name", "unknown");
+            file = MappedFile.mappedFile(OS.TMP + "/.time-stamp." + user + ".dat", OS.pageSize(), 0);
+            IOTools.unmonitor(file);
+            ReferenceOwner mumtp = ReferenceOwner.temporary("mumtp");
+            bytes = file.acquireBytesForWrite(mumtp, 0);
+            bytes.append8bit("&TSF\nTime stamp file uses for sharing a unique id\n");
+            IOTools.unmonitor(bytes);
+        } catch (IOException ioe) {
+            throw new IORuntimeException(ioe);
+        }
+    }
+
+    public MappedUniqueTimeProvider provider(TimeProvider provider) {
+        this.provider = provider;
+        return this;
+    }
+
+    @Override
+    public long currentTimeMillis() {
+        return provider.currentTimeMillis();
+    }
+
+    @Override
+    public long currentTimeMicros() {
+        long timeus = provider.currentTimeMicros();
+        while (true) {
+            long time0 = bytes.readVolatileLong(LAST_TIME);
+            long time0us = time0 / 1000;
+            long time;
+            if (time0us >= timeus)
+                time = (time0us + 1) * 1000;
+            else
+                time = timeus * 1000;
+            if (bytes.compareAndSwapLong(LAST_TIME, time0, time))
+                return time / 1_000;
+            Jvm.nanoPause();
+        }
+    }
+
+    @Override
+    public long currentTimeNanos() {
+        long time = provider.currentTimeNanos();
+        long time5 = time >>> 5;
+
+        long time0 = bytes.readVolatileLong(LAST_TIME);
+        long timeNanos5 = time0 >>> 5;
+
+        if (time5 > timeNanos5)
+            if (bytes.compareAndSwapLong(LAST_TIME, time0, time))
+                return time;
+
+        while (true) {
+            time0 = bytes.readVolatileLong(LAST_TIME);
+            long next = (time0 + 0x20) & ~0x1f;
+            if (bytes.compareAndSwapLong(LAST_TIME, time0, next))
+                return next;
+            Jvm.nanoPause();
+        }
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/MethodEncoder.java

@@ -0,0 +1,28 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+public interface MethodEncoder {
+    long messageId();
+
+    @SuppressWarnings("rawtypes")
+    void encode(Object[] objects, BytesOut out);
+
+    @SuppressWarnings("rawtypes")
+    Object[] decode(Object[] lastObjects, BytesIn in);
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/MethodEncoderLookup.java

@@ -0,0 +1,61 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.util.Annotations;
+
+import java.lang.reflect.Method;
+import java.util.function.Function;
+
+public enum MethodEncoderLookup implements Function<Method, MethodEncoder> {
+    BY_ANNOTATION;
+
+    @Override
+    public MethodEncoder apply(Method method) {
+        MethodId methodId = Annotations.getAnnotation(method, MethodId.class);
+        if (methodId == null) return null;
+        long messageId = methodId.value();
+        return new MethodEncoder() {
+            @Override
+            public long messageId() {
+                return messageId;
+            }
+
+            @SuppressWarnings("rawtypes")
+            @Override
+            public void encode(Object[] objects, BytesOut out) {
+                for (Object object : objects) {
+                    if (object instanceof BytesMarshallable) {
+                        ((BytesMarshallable) object).writeMarshallable(out);
+                        continue;
+                    }
+                    throw new IllegalArgumentException("Object type " + object + " not supported");
+                }
+            }
+
+            @SuppressWarnings("rawtypes")
+            @Override
+            public Object[] decode(Object[] lastObjects, BytesIn in) {
+                for (Object lastObject : lastObjects)
+                    ((BytesMarshallable) lastObject).readMarshallable(in);
+                return lastObjects;
+            }
+        };
+    }
+
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/MethodId.java

@@ -0,0 +1,27 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+import java.lang.annotation.*;
+
+@Retention(RetentionPolicy.RUNTIME)
+@Target(ElementType.METHOD)
+@Inherited
+public @interface MethodId {
+    long value();
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/MethodReader.java

@@ -0,0 +1,53 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.io.Closeable;
+import net.openhft.chronicle.core.util.InvocationTargetRuntimeException;
+
+public interface MethodReader extends Closeable {
+    String HISTORY = "history";
+
+    MethodReaderInterceptorReturns methodReaderInterceptorReturns();
+
+    /**
+     * Moves the queue to read a message if there is one, but is more expensive than {@link #lazyReadOne()}.
+     * If there is an exception in the dispatching mechanics then this should be caught and Jvm.warn'd.
+     * If there is an exception in the invocation then this is wrapped in a {@link InvocationTargetRuntimeException}
+     * and thrown.
+     *
+     * @return true if there was a message, false if no more messages.
+     * @throws InvocationTargetRuntimeException if the receiver (target method) throws
+     */
+    boolean readOne() throws InvocationTargetRuntimeException;
+
+    /**
+     * Does a quick read which is simpler but might not read the next message. readOne() has to be called periodically.
+     * See also javadoc for {@link #readOne()}
+     *
+     * @return true if there was a message, false if there is probably not a message.
+     */
+    default boolean lazyReadOne() {
+        return readOne();
+    }
+
+    /**
+     * Call close on the input when closed
+     */
+    MethodReader closeIn(boolean closeIn);
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/MethodReaderBuilder.java

@@ -0,0 +1,33 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+public interface MethodReaderBuilder {
+    default MethodReaderBuilder methodReaderInterceptor(MethodReaderInterceptor methodReaderInterceptor) {
+        return methodReaderInterceptorReturns((m, o, a, i) -> {
+            methodReaderInterceptor.intercept(m, o, a, i);
+            return null;
+        });
+    }
+
+    MethodReaderBuilder methodReaderInterceptorReturns(MethodReaderInterceptorReturns methodReaderInterceptorReturns);
+
+    MethodReaderBuilder warnMissing(boolean warnMissing);
+
+    MethodReader build(Object... components);
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/MethodReaderInterceptor.java

@@ -0,0 +1,30 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+
+/**
+ * @deprecated Use {@link MethodReaderInterceptorReturns}
+ */
+@Deprecated(/* to be removed in x.22 */)
+@FunctionalInterface
+public interface MethodReaderInterceptor {
+    void intercept(Method m, Object o, Object[] args, Invocation invocation) throws InvocationTargetException;
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/MethodReaderInterceptorReturns.java

@@ -0,0 +1,26 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+
+@FunctionalInterface
+public interface MethodReaderInterceptorReturns {
+    Object intercept(Method m, Object o, Object[] args, Invocation invocation) throws InvocationTargetException;
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/MethodWriterBuilder.java

@@ -0,0 +1,48 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.io.Closeable;
+
+import java.util.function.Supplier;
+
+public interface MethodWriterBuilder<T> extends Supplier<T> {
+
+    MethodWriterBuilder<T> genericEvent(String genericEvent);
+
+    default MethodWriterBuilder<T> metaData(boolean metaData) {
+        return this;
+    }
+
+    @Deprecated(/* to be removed in x.23 */)
+    MethodWriterBuilder<T> useMethodIds(boolean useMethodIds);
+
+    MethodWriterBuilder<T> onClose(Closeable closeable);
+
+    // sourceId enables this, this isn't useful unless it's set.
+    @Deprecated(/* to be removed in x.22 */)
+    MethodWriterBuilder<T> recordHistory(boolean recordHistory);
+
+    default MethodWriterBuilder<T> updateInterceptor(UpdateInterceptor updateInterceptor) {
+        throw new UnsupportedOperationException();
+    }
+
+    default T build() {
+        return get();
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/MethodWriterInterceptor.java

@@ -0,0 +1,55 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+import org.jetbrains.annotations.Nullable;
+
+import java.lang.reflect.Method;
+import java.util.function.BiConsumer;
+
+/**
+ * Invoked around method writing allowing you to take action before or after method invocation,
+ * or even not to call the method
+ *
+ * @deprecated Use MethodWriterInterceptorReturns
+ */
+@FunctionalInterface
+@Deprecated(/* to be removed in x.22 */)
+public interface MethodWriterInterceptor {
+
+    static MethodWriterInterceptor of( final MethodWriterListener methodWriterListener,  final MethodWriterInterceptor interceptor) {
+        if (methodWriterListener == null && interceptor == null)
+            throw new IllegalArgumentException("both methodWriterListener and interceptor are NULL");
+
+        if (methodWriterListener == null)
+            return interceptor::intercept;
+
+        if (interceptor == null)
+            return (method, args, invoker) -> {
+                methodWriterListener.onWrite(method.getName(), args);
+                invoker.accept(method, args);
+            };
+
+        return (method, args, invoker) -> {
+            interceptor.intercept(method, args, invoker);
+            methodWriterListener.onWrite(method.getName(), args);
+        };
+    }
+
+    void intercept(Method method, Object[] args, BiConsumer<Method, Object[]> invoker);
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/MethodWriterInterceptorReturns.java

@@ -0,0 +1,60 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+import org.jetbrains.annotations.Nullable;
+
+import java.lang.reflect.Method;
+import java.util.function.BiFunction;
+
+/*
+ * Peter Lawrey so interceptors can return an object to use for chaining.
+ * <p>
+ * Invoked around method writing allowing you to take action before or after method invocation,
+ * or even not to call the method
+ */
+@FunctionalInterface
+public interface MethodWriterInterceptorReturns {
+    static MethodWriterInterceptorReturns of(MethodWriterInterceptor interceptor) {
+        return (m, a, i) -> {
+            interceptor.intercept(m, a, i::apply);
+            return null;
+        };
+    }
+
+    static MethodWriterInterceptorReturns of( final MethodWriterListener methodWriterListener,  final MethodWriterInterceptorReturns interceptor) {
+        if (methodWriterListener == null && interceptor == null)
+            throw new IllegalArgumentException("both methodWriterListener and interceptor are NULL");
+
+        if (methodWriterListener == null)
+            return interceptor;
+
+        if (interceptor == null)
+            return (method, args, invoker) -> {
+                methodWriterListener.onWrite(method.getName(), args);
+                return invoker.apply(method, args);
+            };
+
+        return (method, args, invoker) -> {
+            methodWriterListener.onWrite(method.getName(), args);
+            return interceptor.intercept(method, args, invoker);
+        };
+    }
+
+    Object intercept(Method method, Object[] args, BiFunction<Method, Object[], Object> invoker);
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/MethodWriterInvocationHandler.java

@@ -0,0 +1,42 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.io.Closeable;
+import org.jetbrains.annotations.Nullable;
+
+import java.lang.reflect.InvocationHandler;
+
+public interface MethodWriterInvocationHandler extends InvocationHandler {
+    void recordHistory(boolean recordHistory);
+
+    void onClose(Closeable closeable);
+
+    default void methodWriterInterceptorReturns(MethodWriterListener methodWriterListener,  MethodWriterInterceptorReturns interceptor) {
+        if (methodWriterListener != null || interceptor != null)
+            methodWriterInterceptorReturns(MethodWriterInterceptorReturns.of(methodWriterListener, interceptor));
+    }
+
+    void methodWriterInterceptorReturns(MethodWriterInterceptorReturns methodWriterInterceptor);
+
+    void genericEvent(String genericEvent);
+
+    void useMethodIds(boolean useMethodIds);
+
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/MethodWriterListener.java

@@ -0,0 +1,29 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+/**
+ * Invoked before writing out this method and args.
+ *
+ * @deprecated Use MethodWriterInterceptorReturns
+ */
+@Deprecated(/* to be removed in x.22 */)
+@FunctionalInterface
+public interface MethodWriterListener {
+    void onWrite(String name, Object[] args);
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/MultiReaderBytesRingBuffer.java

@@ -0,0 +1,37 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+import org.jetbrains.annotations.NotNull;
+
+public interface MultiReaderBytesRingBuffer extends BytesRingBuffer {
+
+
+    default RingBufferReader createReader() {
+        return createReader(0);
+    }
+
+    /**
+     * Create a reader
+     *
+     * @param id of reader as each reader has separate read position etc.
+     * @return reader
+     */
+
+    RingBufferReader createReader(int id);
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/NativeBytes.java

@@ -0,0 +1,409 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.bytes.util.DecoratedBufferOverflowException;
+import net.openhft.chronicle.core.*;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+import java.nio.BufferOverflowException;
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+
+import static net.openhft.chronicle.bytes.NativeBytesStore.nativeStoreWithFixedCapacity;
+import static net.openhft.chronicle.bytes.NoBytesStore.noBytesStore;
+
+/**
+ * Elastic memory accessor which can wrap either a ByteBuffer or malloc'ed memory.
+ * <p>
+ * <p>This class can wrap <i>heap</i> ByteBuffers, called <i>Native</i>Bytes for historical reasons.
+ */
+@SuppressWarnings({"rawtypes", "unchecked"})
+public class NativeBytes<Underlying>
+        extends VanillaBytes<Underlying> {
+    private static final boolean BYTES_GUARDED = Jvm.getBoolean("bytes.guarded");
+    private static boolean s_newGuarded = BYTES_GUARDED;
+    private final long capacity;
+
+    public NativeBytes( final BytesStore store, final long capacity) throws IllegalStateException {
+        super(store, 0, capacity);
+        this.capacity = capacity;
+    }
+
+    public NativeBytes( final BytesStore store) throws IllegalStateException {
+        super(store, 0, store.capacity());
+        capacity = store.capacity();
+    }
+
+    /**
+     * For testing
+     *
+     * @return will new NativeBytes be guarded.
+     */
+    public static boolean areNewGuarded() {
+        return s_newGuarded;
+    }
+
+    /**
+     * turn guarding on/off. Can be enabled by assertion with
+     * <code>
+     * assert NativeBytes.setNewGuarded(true);
+     * </code>
+     *
+     * @param guarded turn on if true
+     */
+    public static boolean setNewGuarded(final boolean guarded) {
+        s_newGuarded = guarded;
+        return true;
+    }
+
+    /**
+     * For testing
+     */
+    public static void resetNewGuarded() {
+        s_newGuarded = BYTES_GUARDED;
+    }
+
+
+    public static NativeBytes<Void> nativeBytes() {
+        try {
+            return NativeBytes.wrapWithNativeBytes(noBytesStore(), Bytes.MAX_CAPACITY);
+        } catch (IllegalStateException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+
+    public static NativeBytes<Void> nativeBytes(final long initialCapacity) throws IllegalArgumentException {
+         final NativeBytesStore<Void> store = nativeStoreWithFixedCapacity(initialCapacity);
+        try {
+            try {
+                return NativeBytes.wrapWithNativeBytes(store, Bytes.MAX_CAPACITY);
+            } finally {
+                store.release(INIT);
+            }
+        } catch (IllegalStateException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+    public static BytesStore<Bytes<Void>, Void> copyOf( final Bytes bytes) {
+        final long remaining = bytes.readRemaining();
+
+        try {
+            final NativeBytes<Void> bytes2 = Bytes.allocateElasticDirect(remaining);
+            bytes2.write(bytes, 0, remaining);
+            return bytes2;
+        } catch (IllegalArgumentException | BufferOverflowException | BufferUnderflowException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+    private static long alignToPageSize(final long size) {
+        long mask = OS.pageSize() - 1;
+        return (size + mask) & ~mask;
+    }
+
+
+    public static <T> NativeBytes<T> wrapWithNativeBytes( final BytesStore<?, T> bs, long capacity) {
+        return s_newGuarded
+                ? new GuardedNativeBytes(bs, capacity)
+                : new NativeBytes<>(bs, capacity);
+    }
+
+    protected static <T> long maxCapacityFor( BytesStore<?, T> bs) {
+        return bs.underlyingObject() instanceof ByteBuffer
+                || bs.underlyingObject() instanceof byte[]
+                ? MAX_HEAP_CAPACITY
+                : Bytes.MAX_CAPACITY;
+    }
+
+    @Override
+    public long capacity() {
+        return capacity;
+    }
+
+    @Override
+    protected void writeCheckOffset(final long offset, final long adding) throws BufferOverflowException {
+        if (offset >= bytesStore.start()) {
+            final long writeEnd = offset + adding;
+            if (writeEnd <= bytesStore.safeLimit()) {
+                return; // do nothing.
+            }
+            if (writeEnd >= capacity)
+                throw new BufferOverflowException(/*"Write exceeds capacity"*/);
+            checkResize(writeEnd);
+        } else {
+            throw new BufferOverflowException();
+        }
+    }
+
+    @Override
+    void prewriteCheckOffset(long offset, long subtracting) throws BufferOverflowException {
+        if (offset - subtracting >= bytesStore.start()) {
+            if (offset <= bytesStore.safeLimit()) {
+                return; // do nothing.
+            }
+            if (offset >= capacity)
+                throw new BufferOverflowException(/*"Write exceeds capacity"*/);
+            checkResize(offset);
+        } else {
+            throw new BufferOverflowException();
+        }
+    }
+
+    @Override
+    public void ensureCapacity(final long size) throws IllegalArgumentException {
+        try {
+            assert size >= 0;
+            writeCheckOffset(writePosition(), size);
+        } catch (BufferOverflowException e) {
+            IllegalArgumentException iae = new IllegalArgumentException("Bytes cannot be resized to " + size + " limit: " + capacity());
+            iae.printStackTrace();
+            throw iae;
+        }
+    }
+
+    private void checkResize(final long endOfBuffer) throws BufferOverflowException {
+        if (isElastic())
+            resize(endOfBuffer);
+        else
+            throw new BufferOverflowException();
+    }
+
+    @Override
+    public boolean isElastic() {
+        return true;
+    }
+
+    // the endOfBuffer is the minimum capacity and one byte more than the last addressable byte.
+    private void resize(final long endOfBuffer)
+            throws BufferOverflowException {
+        throwExceptionIfReleased();
+        if (endOfBuffer < 0)
+            throw new DecoratedBufferOverflowException(endOfBuffer + "< 0");
+        if (endOfBuffer > capacity())
+            throw new DecoratedBufferOverflowException(endOfBuffer + ">" + capacity());
+        final long realCapacity = realCapacity();
+        if (endOfBuffer <= realCapacity) {
+//            System.out.println("no resize " + endOfBuffer + " < " + realCapacity);
+            return;
+        }
+
+        // Grow by 50%
+        long size = Math.max(endOfBuffer + 7, realCapacity * 3 / 2 + 32);
+        if (isDirectMemory() || size > MAX_HEAP_CAPACITY) {
+            // Allocate direct memory of page granularity
+            size = alignToPageSize(size);
+        } else {
+            size &= ~0x7;
+        }
+        // Cap the size with capacity() again
+        size = Math.min(size, capacity());
+
+        final boolean isByteBufferBacked = bytesStore.underlyingObject() instanceof ByteBuffer;
+        if (isByteBufferBacked && size > MAX_HEAP_CAPACITY) {
+
+            // Add a stack trace to this relatively unusual event which will
+            // enable tracing of potentially derailed code or excessive buffer use.
+            final StackTrace stackTrace = new StackTrace();
+            final String stack = BytesUtil.asString("Calling stack is", stackTrace);
+
+            Jvm.warn().on(getClass(), "Going to try to replace ByteBuffer-backed BytesStore with " +
+                    "raw NativeBytesStore to grow to " + size / 1024 + " KB. If later it is assumed that " +
+                    "this bytes' underlyingObject() is ByteBuffer, NullPointerException is likely to be thrown. " +
+                    stack);
+        }
+//        System.out.println("resize " + endOfBuffer + " to " + size);
+        if (endOfBuffer > 1 << 20)
+            Jvm.warn().on(getClass(), "Resizing buffer was " + realCapacity / 1024 + " KB, " +
+                    "needs " + (endOfBuffer - realCapacity) + " bytes more, " +
+                    "new-size " + size / 1024 + " KB");
+        final BytesStore store;
+        int position = 0;
+        try {
+            if (isByteBufferBacked && size <= MAX_HEAP_CAPACITY) {
+                position = ((ByteBuffer) bytesStore.underlyingObject()).position();
+                store = allocateNewByteBufferBackedStore(Maths.toInt32(size));
+            } else {
+                store = NativeBytesStore.lazyNativeBytesStoreWithFixedCapacity(size);
+            }
+            store.reserveTransfer(INIT, this);
+        } catch (IllegalArgumentException e) {
+            BufferOverflowException boe = new BufferOverflowException();
+            boe.initCause(e);
+            throw boe;
+        }
+
+        throwExceptionIfReleased();
+         final BytesStore<Bytes<Underlying>, Underlying> tempStore = this.bytesStore;
+        this.bytesStore.copyTo(store);
+        this.bytesStore(store);
+        try {
+            tempStore.release(this);
+        } catch (IllegalStateException e) {
+            Jvm.debug().on(getClass(), e);
+        }
+
+        if (this.bytesStore.underlyingObject() instanceof ByteBuffer) {
+             final ByteBuffer byteBuffer = (ByteBuffer) this.bytesStore.underlyingObject();
+            byteBuffer.position(0);
+            byteBuffer.limit(byteBuffer.capacity());
+            byteBuffer.position(position);
+        }
+    }
+
+
+    private BytesStore allocateNewByteBufferBackedStore(final int size) {
+        if (isDirectMemory()) {
+            return NativeBytesStore.elasticByteBuffer(size, capacity());
+        } else {
+            return HeapBytesStore.wrap(ByteBuffer.allocate(size));
+        }
+    }
+
+
+    @Override
+    public Bytes<Underlying> write( final byte[] bytes,
+                                   final int offset,
+                                   final int length) throws BufferOverflowException, IllegalArgumentException {
+        if (length > writeRemaining())
+            throw new BufferOverflowException();
+        ensureCapacity(length);
+        super.write(bytes, offset, length);
+        return this;
+    }
+
+
+    public Bytes<Underlying> write( final BytesStore bytes,
+                                   final long offset,
+                                   final long length) throws BufferOverflowException, IllegalArgumentException, BufferUnderflowException {
+        ensureCapacity(length);
+        super.write(bytes, offset, length);
+        return this;
+    }
+
+    @Override
+
+    public NativeBytes writeSome( final Bytes bytes) {
+        try {
+            long length = Math.min(bytes.readRemaining(), writeRemaining());
+            if (length + writePosition() >= 1 << 20)
+                length = Math.min(bytes.readRemaining(), realCapacity() - writePosition());
+            final long offset = bytes.readPosition();
+            ensureCapacity(length);
+            optimisedWrite(bytes, offset, length);
+            if (length == bytes.readRemaining()) {
+                bytes.clear();
+            } else {
+                bytes.readSkip(length);
+                if (bytes.writePosition() > bytes.realCapacity() / 2)
+                    bytes.compact();
+            }
+            return this;
+        } catch (IllegalArgumentException | BufferUnderflowException | BufferOverflowException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+    @Override
+    protected long writeOffsetPositionMoved(final long adding, final long advance) throws BufferOverflowException {
+        final long oldPosition = writePosition;
+        if (writePosition < bytesStore.start())
+            throw new BufferOverflowException();
+        final long writeEnd = writePosition + adding;
+        if (writeEnd > writeLimit)
+            throwBeyondWriteLimit(advance, writeEnd);
+        else if (writeEnd > bytesStore.safeLimit())
+            checkResize(writeEnd);
+        this.writePosition = writePosition + advance;
+        return oldPosition;
+    }
+
+    private void throwBeyondWriteLimit(long advance, long writeEnd) {
+        throw new DecoratedBufferOverflowException("attempt to write " + advance + " bytes to " + writeEnd + " limit: " + writeLimit);
+    }
+
+
+    @Override
+    public Bytes<Underlying> writeByte(final byte i8) throws BufferOverflowException {
+        final long offset = writeOffsetPositionMoved(1, 1);
+        bytesStore.writeByte(offset, i8);
+        return this;
+    }
+
+
+    @Override
+    public Bytes<Underlying> write8bit( final BytesStore bs) throws BufferOverflowException {
+        if (bs == null) {
+            writeStopBit(-1);
+        } else {
+            final long offset = bs.readPosition();
+            final long readRemaining = Math.min(writeRemaining(), bs.readLimit() - offset);
+            writeStopBit(readRemaining);
+            write(bs, offset, readRemaining);
+        }
+        return this;
+    }
+
+
+    @Override
+    public Bytes<Underlying> writeLong(final long i64) throws BufferOverflowException {
+        final long offset = writeOffsetPositionMoved(8L, 8L);
+        bytesStore.writeLong(offset, i64);
+        return this;
+    }
+
+    @Override
+    public long readRemaining() {
+        return writePosition - readPosition;
+    }
+
+    public final static class NativeSubBytes extends SubBytes {
+        private final NativeBytesStore nativeBytesStore;
+
+        public NativeSubBytes( final BytesStore bytesStore,
+                              final long start,
+                              final long capacity) throws IllegalStateException {
+            super(bytesStore, start, capacity);
+            nativeBytesStore = (NativeBytesStore) this.bytesStore;
+        }
+
+        @Override
+        public long read(final long offsetInRDI,
+                          final byte[] bytes,
+                         final int offset,
+                         final int length) {
+
+            final int len = (int) Math.min(length, readLimit() - offsetInRDI);
+            int i;
+            final long address = nativeBytesStore.address + nativeBytesStore.translate(offsetInRDI);
+            for (i = 0; i < len - 7; i += 8)
+                UnsafeMemory.unsafePutLong(bytes, i, nativeBytesStore.memory.readLong(address + i));
+            if (i < len - 3) {
+                UnsafeMemory.unsafePutInt(bytes, i, nativeBytesStore.memory.readInt(address + i));
+                i += 4;
+            }
+            for (; i < len; i++)
+                UnsafeMemory.unsafePutByte(bytes, i, nativeBytesStore.memory.readByte(address + i));
+            return len;
+        }
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/NativeBytesStore.java

@@ -0,0 +1,852 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.*;
+import net.openhft.chronicle.core.annotation.ForceInline;
+import net.openhft.chronicle.core.cleaner.CleanerServiceLocator;
+import net.openhft.chronicle.core.cleaner.spi.ByteBufferCleanerService;
+import net.openhft.chronicle.core.io.IORuntimeException;
+import net.openhft.chronicle.core.util.SimpleCleaner;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+import java.lang.reflect.Field;
+import java.nio.BufferOverflowException;
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+
+import static net.openhft.chronicle.bytes.Bytes.MAX_CAPACITY;
+
+@SuppressWarnings({"restriction", "rawtypes", "unchecked"})
+public class NativeBytesStore<Underlying>
+        extends AbstractBytesStore<NativeBytesStore<Underlying>, Underlying> {
+    private static final long MEMORY_MAPPED_SIZE = 128 << 10;
+    private static final Field BB_ADDRESS, BB_CAPACITY, BB_ATT;
+    private static final ByteBufferCleanerService CLEANER_SERVICE = CleanerServiceLocator.cleanerService();
+    //    static MappedBytes last;
+
+    static {
+        Class directBB = ByteBuffer.allocateDirect(0).getClass();
+        BB_ADDRESS = Jvm.getField(directBB, "address");
+        BB_CAPACITY = Jvm.getField(directBB, "capacity");
+        BB_ATT = Jvm.getField(directBB, "att");
+    }
+
+    protected long address;
+    // on release, set this to null.
+    protected Memory memory = OS.memory();
+    protected long limit, maximumLimit;
+
+    private SimpleCleaner cleaner;
+    private boolean elastic;
+
+    private Underlying underlyingObject;
+    private final Finalizer finalizer;
+
+    private NativeBytesStore() {
+        finalizer = null;
+    }
+
+    private NativeBytesStore( ByteBuffer bb, boolean elastic) {
+        this(bb, elastic, Bytes.MAX_HEAP_CAPACITY);
+    }
+
+    public NativeBytesStore( ByteBuffer bb, boolean elastic, int maximumLimit) {
+        this();
+        init(bb, elastic);
+        this.maximumLimit = elastic ? maximumLimit : Math.min(limit, maximumLimit);
+    }
+
+    public NativeBytesStore(
+            long address, long limit) {
+        this(address, limit, null, false);
+    }
+
+    public NativeBytesStore(
+            long address, long limit,  Runnable deallocator, boolean elastic) {
+        this(address, limit, deallocator, elastic, false);
+    }
+
+    protected NativeBytesStore(
+            long address, long limit,  Runnable deallocator, boolean elastic, boolean monitored) {
+        super(monitored);
+        setAddress(address);
+        this.limit = limit;
+        this.maximumLimit = elastic ? MAX_CAPACITY : limit;
+        this.cleaner = deallocator == null ? null : new SimpleCleaner(deallocator);
+        underlyingObject = null;
+        this.elastic = elastic;
+        if (cleaner == null)
+            finalizer = null;
+        else
+            finalizer = new Finalizer();
+    }
+
+
+    public static NativeBytesStore<ByteBuffer> wrap( ByteBuffer bb) {
+        return new NativeBytesStore<>(bb, false);
+    }
+
+
+    public static <T> NativeBytesStore<T> uninitialized() {
+        return new NativeBytesStore<>();
+    }
+
+    /**
+     * this is an elastic native store
+     *
+     * @param capacity of the buffer.
+     */
+
+    public static NativeBytesStore<Void> nativeStore(long capacity)
+            throws IllegalArgumentException {
+        return of(capacity, true, true);
+    }
+
+
+    private static NativeBytesStore<Void> of(long capacity, boolean zeroOut, boolean elastic)
+            throws IllegalArgumentException {
+        if (capacity <= 0)
+            return new NativeBytesStore<>(NoBytesStore.NO_PAGE, 0, null, elastic);
+        Memory memory = OS.memory();
+        long address = memory.allocate(capacity);
+        if (zeroOut || capacity < MEMORY_MAPPED_SIZE) {
+            memory.setMemory(address, capacity, (byte) 0);
+            memory.storeFence();
+        }
+         Deallocator deallocator = new Deallocator(address, capacity);
+        return new NativeBytesStore<>(address, capacity, deallocator, elastic);
+    }
+
+
+    public static NativeBytesStore<Void> nativeStoreWithFixedCapacity(long capacity)
+            throws IllegalArgumentException {
+        return of(capacity, true, false);
+    }
+
+
+    public static NativeBytesStore<Void> lazyNativeBytesStoreWithFixedCapacity(long capacity)
+            throws IllegalArgumentException {
+        return of(capacity, false, false);
+    }
+
+
+    public static NativeBytesStore<ByteBuffer> elasticByteBuffer() {
+        return elasticByteBuffer(OS.pageSize(), MAX_CAPACITY);
+    }
+
+
+    public static NativeBytesStore<ByteBuffer> elasticByteBuffer(int size, long maxSize) {
+        return new NativeBytesStore<>(ByteBuffer.allocateDirect(size), true, Math.toIntExact(maxSize));
+    }
+
+
+    public static NativeBytesStore from( String text) {
+        return from(text.getBytes(StandardCharsets.ISO_8859_1));
+    }
+
+
+    public static NativeBytesStore from( byte[] bytes) {
+        try {
+             NativeBytesStore nbs = nativeStoreWithFixedCapacity(bytes.length);
+            Bytes<byte[]> bytes2 = Bytes.wrapForRead(bytes);
+            bytes2.copyTo(nbs);
+            bytes2.releaseLast();
+            return nbs;
+        } catch (IllegalArgumentException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+    @Override
+    public boolean isDirectMemory() {
+        return true;
+    }
+
+    @Override
+    public boolean canReadDirect(long length) {
+        return limit >= length;
+    }
+
+    public void init( ByteBuffer bb, boolean elastic) {
+        this.elastic = elastic;
+        underlyingObject = (Underlying) bb;
+        setAddress(Jvm.address(bb));
+        this.limit = bb.capacity();
+    }
+
+    public void uninit() {
+        underlyingObject = null;
+        address = 0;
+        limit = 0;
+        maximumLimit = 0;
+        cleaner = null;
+    }
+
+    @Override
+    public void move(long from, long to, long length) throws BufferUnderflowException {
+        if (from < 0 || to < 0) throw new BufferUnderflowException();
+        long address = this.address;
+        if (address == 0) throwException(null);
+        memoryCopyMemory(address + from, address + to, length);
+    }
+
+    private void memoryCopyMemory(long fromAddress, long toAddress, long length) {
+        try {
+            memory.copyMemory(fromAddress, toAddress, length);
+        } catch (NullPointerException ifReleased) {
+            throwException(ifReleased);
+        }
+    }
+
+    private void throwException(Throwable ifReleased) {
+        throwExceptionIfReleased();
+        throw new IllegalStateException(ifReleased);
+    }
+
+
+    @Override
+    public BytesStore<NativeBytesStore<Underlying>, Underlying> copy() {
+        try {
+            if (underlyingObject == null) {
+                 NativeBytesStore<Void> copy = of(realCapacity(), false, true);
+                memoryCopyMemory(address, copy.address, capacity());
+                return (BytesStore) copy;
+
+            } else if (underlyingObject instanceof ByteBuffer) {
+                ByteBuffer bb = ByteBuffer.allocateDirect(Maths.toInt32(capacity()));
+                bb.put((ByteBuffer) underlyingObject);
+                bb.clear();
+                return (BytesStore) wrap(bb);
+
+            } else {
+                throw new UnsupportedOperationException();
+            }
+        } catch (IllegalArgumentException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+
+    @Override
+    public VanillaBytes<Underlying> bytesForWrite() throws IllegalStateException {
+        return elastic
+                ? NativeBytes.wrapWithNativeBytes(this, this.capacity())
+                : new VanillaBytes<>(this);
+    }
+
+    @Override
+    @ForceInline
+    public long realCapacity() {
+        return limit;
+    }
+
+    @Override
+    @ForceInline
+    public long capacity() {
+        return maximumLimit;
+    }
+
+
+    @Override
+    @ForceInline
+    public Underlying underlyingObject() {
+        return underlyingObject;
+    }
+
+
+    @Override
+    @ForceInline
+    public NativeBytesStore<Underlying> zeroOut(long start, long end) {
+        if (end <= start)
+            return this;
+        if (start < start())
+            start = start();
+        if (end > capacity())
+            end = capacity();
+
+        memory.setMemory(address + translate(start), end - start, (byte) 0);
+        return this;
+    }
+
+    @Override
+    @ForceInline
+    public boolean compareAndSwapInt(long offset, int expected, int value) {
+        return memory.compareAndSwapInt(address + translate(offset), expected, value);
+    }
+
+    @Override
+    public void testAndSetInt(long offset, int expected, int value) {
+        memory.testAndSetInt(address + translate(offset), offset, expected, value);
+    }
+
+    @Override
+    @ForceInline
+    public boolean compareAndSwapLong(long offset, long expected, long value) {
+        return memory.compareAndSwapLong(address + translate(offset), expected, value);
+    }
+
+    @Override
+    @ForceInline
+    public long addAndGetLong(long offset, long adding) throws BufferUnderflowException {
+        return memory.addLong(address + translate(offset), adding);
+    }
+
+    @Override
+    @ForceInline
+    public int addAndGetInt(long offset, int adding) throws BufferUnderflowException {
+        return memory.addInt(address + translate(offset), adding);
+    }
+
+    protected long translate(long offset) {
+        return offset;
+    }
+
+    @Override
+    @ForceInline
+    public byte readByte(long offset) {
+        return memory.readByte(address + translate(offset));
+    }
+
+    public int readUnsignedByte(long offset) throws BufferUnderflowException {
+        return readByte(offset) & 0xFF;
+    }
+
+    @Override
+    @ForceInline
+    public short readShort(long offset) {
+        return memory.readShort(address + translate(offset));
+    }
+
+    @Override
+    @ForceInline
+    public int readInt(long offset) {
+        return memory.readInt(address + translate(offset));
+    }
+
+    @Override
+    @ForceInline
+    public long readLong(long offset) {
+        long address = this.address;
+        assert address != 0;
+        return memory.readLong(address + translate(offset));
+    }
+
+    @Override
+    @ForceInline
+    public float readFloat(long offset) {
+        return memory.readFloat(address + translate(offset));
+    }
+
+    @Override
+    @ForceInline
+    public double readDouble(long offset) {
+        return memory.readDouble(address + translate(offset));
+    }
+
+    @Override
+    @ForceInline
+    public byte readVolatileByte(long offset) {
+        return memory.readVolatileByte(address + translate(offset));
+    }
+
+    @Override
+    @ForceInline
+    public short readVolatileShort(long offset) {
+        return memory.readVolatileShort(address + translate(offset));
+    }
+
+    @Override
+    @ForceInline
+    public int readVolatileInt(long offset) {
+        return memory.readVolatileInt(address + translate(offset));
+    }
+
+    @Override
+    @ForceInline
+    public long readVolatileLong(long offset) {
+        return memory.readVolatileLong(address + translate(offset));
+    }
+
+
+    @Override
+    @ForceInline
+    public NativeBytesStore<Underlying> writeByte(long offset, byte i8) {
+        memory.writeByte(address + translate(offset), i8);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public NativeBytesStore<Underlying> writeShort(long offset, short i16) {
+        memory.writeShort(address + translate(offset), i16);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public NativeBytesStore<Underlying> writeInt(long offset, int i32) {
+        try {
+            memory.writeInt(address + translate(offset), i32);
+            return this;
+        } catch (NullPointerException e) {
+            throwExceptionIfReleased();
+            throw e;
+        }
+    }
+
+
+    @Override
+    @ForceInline
+    public NativeBytesStore<Underlying> writeOrderedInt(long offset, int i) {
+        memory.writeOrderedInt(address + translate(offset), i);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public NativeBytesStore<Underlying> writeLong(long offset, long i64) {
+        memory.writeLong(address + translate(offset), i64);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public NativeBytesStore<Underlying> writeOrderedLong(long offset, long i) {
+        memory.writeOrderedLong(address + translate(offset), i);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public NativeBytesStore<Underlying> writeFloat(long offset, float f) {
+        memory.writeFloat(address + translate(offset), f);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public NativeBytesStore<Underlying> writeDouble(long offset, double d) {
+        memory.writeDouble(address + translate(offset), d);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public NativeBytesStore<Underlying> writeVolatileByte(long offset, byte i8) {
+        memory.writeVolatileByte(address + translate(offset), i8);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public NativeBytesStore<Underlying> writeVolatileShort(long offset, short i16) {
+        memory.writeVolatileShort(address + translate(offset), i16);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public NativeBytesStore<Underlying> writeVolatileInt(long offset, int i32) {
+        memory.writeVolatileInt(address + translate(offset), i32);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public NativeBytesStore<Underlying> writeVolatileLong(long offset, long i64) {
+        memory.writeVolatileLong(address + translate(offset), i64);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public NativeBytesStore<Underlying> write(
+            long offsetInRDO, byte[] bytes, int offset, int length) {
+        memory.copyMemory(bytes, offset, address + translate(offsetInRDO), length);
+        return this;
+    }
+
+    @Override
+    @ForceInline
+    public void write(
+            long offsetInRDO,  ByteBuffer bytes, int offset, int length) {
+        if (bytes.isDirect()) {
+            memoryCopyMemory(Jvm.address(bytes) + offset, address + translate(offsetInRDO), length);
+
+        } else {
+            memory.copyMemory(bytes.array(), offset, address + translate(offsetInRDO), length);
+        }
+    }
+
+
+    @Override
+    @ForceInline
+    public NativeBytesStore<Underlying> write(
+            long writeOffset,  RandomDataInput bytes, long readOffset, long length)
+            throws BufferOverflowException, BufferUnderflowException {
+        if (bytes.isDirectMemory()) {
+            memoryCopyMemory(bytes.addressForRead(readOffset), addressForWrite(writeOffset), length);
+        } else {
+            write0(writeOffset, bytes, readOffset, length);
+        }
+        return this;
+    }
+
+    public void write0(long offsetInRDO,  RandomDataInput bytes, long offset, long length) throws BufferUnderflowException {
+        long i = 0;
+        for (; i < length - 7; i += 8) {
+            writeLong(offsetInRDO + i, bytes.readLong(offset + i));
+        }
+        for (; i < length; i++) {
+            writeByte(offsetInRDO + i, bytes.readByte(offset + i));
+        }
+    }
+
+    @Override
+    public long addressForRead(long offset) throws BufferUnderflowException {
+        if (offset < start() || offset > realCapacity())
+            throw new BufferUnderflowException();
+        return address + translate(offset);
+    }
+
+    @Override
+    public long addressForWrite(long offset) throws BufferOverflowException {
+        if (offset < start() || offset > realCapacity())
+            throw new BufferOverflowException();
+        return address + translate(offset);
+    }
+
+    @Override
+    public long addressForWritePosition() throws UnsupportedOperationException, BufferOverflowException {
+        return addressForWrite(start());
+    }
+
+    @Override
+    protected void performRelease() {
+        memory = null;
+        if (cleaner != null) {
+            cleaner.clean();
+
+        } else if (underlyingObject instanceof ByteBuffer) {
+            ByteBuffer underlyingObject = (ByteBuffer) this.underlyingObject;
+            if (underlyingObject.isDirect())
+                CLEANER_SERVICE.clean(underlyingObject);
+        }
+    }
+
+
+    @Override
+    public String toString() {
+        try {
+            return BytesInternal.toString(this);
+
+        } catch (IllegalStateException e) {
+            return e.toString();
+        }
+    }
+
+    @Override
+    @ForceInline
+    public void nativeRead(long position, long address, long size) throws BufferUnderflowException {
+        // TODO add bounds checking.
+        memoryCopyMemory(addressForRead(position), address, size);
+    }
+
+    @Override
+    @ForceInline
+    public void nativeWrite(long address, long position, long size) throws BufferOverflowException {
+        // TODO add bounds checking.
+        memoryCopyMemory(address, addressForWrite(position), size);
+    }
+
+    void write8bit(long position, char[] chars, int offset, int length) {
+        long addr = address + translate(position);
+         Memory memory = this.memory;
+        for (int i = 0; i < length; i++)
+            memory.writeByte(addr + i, (byte) chars[offset + i]);
+    }
+
+    void read8bit(long position, char[] chars, int length) {
+        long addr = address + translate(position);
+        Memory memory = this.memory;
+        for (int i = 0; i < length; i++)
+            chars[i] = (char) (memory.readByte(addr + i) & 0xFF);
+    }
+
+    @Override
+    public long readIncompleteLong(long offset) {
+        int remaining = (int) Math.min(8, readRemaining() - offset);
+        long l = 0;
+        for (int i = 0; i < remaining; i++) {
+            byte b = memory.readByte(address + offset + i);
+            l |= (long) (b & 0xFF) << (i * 8);
+        }
+        return l;
+    }
+
+    @Override
+    public boolean equals(Object obj) {
+        return obj instanceof BytesStore && BytesInternal.contentEqual(this, (BytesStore) obj);
+    }
+
+    public void setAddress(long address) {
+        if ((address & ~0x3FFF) == 0)
+            throw new AssertionError("Invalid addressForRead " + Long.toHexString(address));
+        this.address = address;
+    }
+
+    @Deprecated(/* to be removed in x.22 */)
+    public long appendUTF(long pos, char[] chars, int offset, int length) throws BufferOverflowException {
+        return appendUtf8(pos, chars, offset, length);
+    }
+
+    public long appendUtf8(long pos, char[] chars, int offset, int length) throws BufferOverflowException {
+        if (pos + length > realCapacity())
+            throw new BufferOverflowException();
+
+        long address = this.address + translate(0);
+         Memory memory = this.memory;
+        if (memory == null) throw new NullPointerException();
+        int i;
+        ascii:
+        {
+            for (i = 0; i < length - 3; i += 4) {
+                final int i2 = offset + i;
+                char c0 = chars[i2];
+                char c1 = chars[i2 + 1];
+                char c2 = chars[i2 + 2];
+                char c3 = chars[i2 + 3];
+                if ((c0 | c1 | c2 | c3) > 0x007F)
+                    break ascii;
+                final int value = (c0) | (c1 << 8) | (c2 << 16) | (c3 << 24);
+                UnsafeMemory.unsafePutInt(address + pos, value);
+                pos += 4;
+            }
+            for (; i < length; i++) {
+                char c = chars[offset + i];
+                if (c > 0x007F)
+                    break ascii;
+                UnsafeMemory.unsafePutByte(address + pos++, (byte) c);
+            }
+
+            return pos;
+        }
+        return appendUtf8a(pos, chars, offset, length, i);
+    }
+
+    private long appendUtf8a(long pos, char[] chars, int offset, int length, int i) {
+        for (; i < length; i++) {
+            char c = chars[offset + i];
+            if (c <= 0x007F) {
+                writeByte(pos++, (byte) c);
+
+            } else if (c <= 0x07FF) {
+                writeByte(pos++, (byte) (0xC0 | ((c >> 6) & 0x1F)));
+                writeByte(pos++, (byte) (0x80 | c & 0x3F));
+
+            } else {
+                writeByte(pos++, (byte) (0xE0 | ((c >> 12) & 0x0F)));
+                writeByte(pos++, (byte) (0x80 | ((c >> 6) & 0x3F)));
+                writeByte(pos++, (byte) (0x80 | (c & 0x3F)));
+            }
+        }
+        return pos;
+    }
+
+    @Override
+    public long copyTo( BytesStore store) {
+        if (store.isDirectMemory())
+            return copyToDirect(store);
+        else
+            return super.copyTo(store);
+    }
+
+    public long copyToDirect( BytesStore store) {
+        long toCopy = Math.min(limit, store.safeLimit());
+        if (toCopy > 0) {
+            try {
+                long addr = address;
+                long addr2 = store.addressForWrite(0);
+                memoryCopyMemory(addr, addr2, toCopy);
+            } catch (BufferOverflowException e) {
+                throw new AssertionError(e);
+            }
+        }
+        return toCopy;
+    }
+
+
+    @Override
+    public ByteBuffer toTemporaryDirectByteBuffer() {
+        ByteBuffer bb = ByteBuffer.allocateDirect(0);
+        try {
+            BB_ADDRESS.setLong(bb, address);
+            BB_CAPACITY.setInt(bb, Maths.toUInt31(readRemaining()));
+            BB_ATT.set(bb, this);
+        } catch (Exception e) {
+            throw new AssertionError(e);
+        }
+        bb.clear();
+        return bb;
+    }
+
+    @Override
+    public int byteCheckSum() throws IORuntimeException {
+        return byteCheckSum(start(), readLimit());
+    }
+
+    public int byteCheckSum(long position, long limit) {
+         Memory memory = this.memory;
+        assert memory != null;
+        int b = 0;
+        long ptr = address + position;
+        long end = address + limit;
+        for (; ptr < end - 7; ptr += 8) {
+            b += memory.readByte(ptr)
+                    + memory.readByte(ptr + 1)
+                    + memory.readByte(ptr + 2)
+                    + memory.readByte(ptr + 3)
+                    + memory.readByte(ptr + 4)
+                    + memory.readByte(ptr + 5)
+                    + memory.readByte(ptr + 6)
+                    + memory.readByte(ptr + 7);
+        }
+        for (; ptr < end; ptr++) {
+            b += memory.readByte(ptr);
+        }
+        return b & 0xFF;
+    }
+
+    @Override
+    public boolean sharedMemory() {
+        return false;
+    }
+
+    @Override
+    public long read(long offsetInRDI, byte[] bytes, int offset, int length) {
+        int len = (int) Math.min(length, readLimit() - offsetInRDI);
+        int i;
+        final long address = this.address + translate(offsetInRDI);
+        for (i = 0; i < len - 7; i += 8)
+            UnsafeMemory.unsafePutLong(bytes, i, memory.readLong(address + i));
+        if (i < len - 3) {
+            UnsafeMemory.unsafePutInt(bytes, i, memory.readInt(address + i));
+            i += 4;
+        }
+        for (; i < len; i++)
+            UnsafeMemory.unsafePutByte(bytes, i, memory.readByte(address + i));
+        return len;
+    }
+
+    @Override
+    public int peekUnsignedByte(long offset) {
+        final long address = this.address;
+         final Memory memory = this.memory;
+        final long translate = translate(offset);
+//        assert translate >= 0;
+        final long address2 = address + translate;
+//        last.writeLong(8, Thread.currentThread().getId());
+//        last.writeLong(0, offset);
+//        last.writeLong(16, translate);
+//        last.writeLong(32, maximumLimit);
+//        last.writeLong(48, addressForRead);
+//        last.writeLong(64, address2);
+//        last.writeBoolean(80, memory != null);
+//        last.writeVolatileByte(88, (byte) 1);
+        int ret = translate >= limit ? -1 :
+                memory.readByte(address2) & 0xFF;
+//        last.writeVolatileByte(88, (byte) 0xFF);
+//        last.writeLong(24, Thread.currentThread().getId());
+        return ret;
+    }
+
+    @Override
+    public int fastHash(long offset, int length) throws BufferUnderflowException {
+        long ret;
+        switch (length) {
+            case 0:
+                return 0;
+            case 1:
+                ret = readByte(offset);
+                break;
+            case 2:
+                ret = readShort(offset);
+                break;
+            case 4:
+                ret = readInt(offset);
+                break;
+            case 8:
+                ret = readInt(offset) * 0x6d0f27bdL + readInt(offset + 4);
+                break;
+            default:
+                return super.fastHash(offset, length);
+        }
+        long hash = ret * 0x855dd4db;
+        return (int) (hash ^ (hash >> 32));
+    }
+
+    @Override
+    public long safeLimit() {
+        return limit;
+    }
+
+    static class Deallocator implements Runnable {
+
+        private volatile long address;
+        private final long size;
+
+        Deallocator(long address, long size) {
+            assert address != 0;
+            this.address = address;
+            this.size = size;
+        }
+
+        @Override
+        public void run() {
+            //     System.out.println("Release " + Long.toHexString(addressForRead));
+            if (address == 0)
+                return;
+            long addressToFree = address;
+            address = 0;
+            OS.memory().freeMemory(addressToFree, size);
+        }
+    }
+
+    private class Finalizer {
+        @Override
+        protected void finalize() throws Throwable {
+            super.finalize();
+            warnAndReleaseIfNotReleased();
+        }
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/NewChunkListener.java

@@ -0,0 +1,24 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+@FunctionalInterface
+public interface NewChunkListener {
+    void onNewChunk(String filename, int chunk, long delayMicros);
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/NoBytesStore.java

@@ -0,0 +1,338 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.OS;
+import net.openhft.chronicle.core.io.IOTools;
+import net.openhft.chronicle.core.io.ReferenceOwner;
+import org.jetbrains.annotations.NotNull;
+
+import java.nio.BufferOverflowException;
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+
+/**
+ * This is a ByteStore which uses no space but could be resized to be larger (by replacing it with a ByteStire with space)
+ *
+ * @see ReleasedBytesStore
+ */
+@SuppressWarnings({"rawtypes", "unchecked"})
+public enum NoBytesStore implements BytesStore {
+    NO_BYTES_STORE;
+
+    public static final long NO_PAGE;
+
+
+    public static final Bytes NO_BYTES;
+
+    static {
+        try {
+            NO_PAGE = OS.memory().allocate(OS.pageSize());
+            NO_BYTES = new VanillaBytes(noBytesStore());
+            IOTools.unmonitor(NO_BYTES);
+
+        } catch ( IllegalArgumentException | IllegalStateException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+
+    public static <T, B extends BytesStore<B, T>> BytesStore<B, T> noBytesStore() {
+        return NO_BYTES_STORE;
+    }
+
+    @Override
+    public void reserve(ReferenceOwner owner) throws IllegalStateException {
+    }
+
+    @Override
+    public void release(ReferenceOwner owner) throws IllegalStateException {
+    }
+
+    @Override
+    public void releaseLast(ReferenceOwner id) throws IllegalStateException {
+    }
+
+    @Override
+    public int refCount() {
+        return 1;
+    }
+
+    @Override
+    public boolean tryReserve(ReferenceOwner owner) {
+        return false;
+    }
+
+    @Override
+    public boolean reservedBy(ReferenceOwner owner) {
+        return true;
+    }
+
+
+    @Override
+    public RandomDataOutput writeByte(long offset, byte i8) {
+        throw new UnsupportedOperationException();
+    }
+
+
+    @Override
+    public RandomDataOutput writeShort(long offset, short i) {
+        throw new UnsupportedOperationException();
+    }
+
+
+    @Override
+    public RandomDataOutput writeInt(long offset, int i) {
+        throw new UnsupportedOperationException();
+    }
+
+
+    @Override
+    public RandomDataOutput writeOrderedInt(long offset, int i) {
+        throw new UnsupportedOperationException();
+    }
+
+
+    @Override
+    public RandomDataOutput writeLong(long offset, long i) {
+        throw new UnsupportedOperationException();
+    }
+
+
+    @Override
+    public RandomDataOutput writeOrderedLong(long offset, long i) {
+        throw new UnsupportedOperationException();
+    }
+
+
+    @Override
+    public RandomDataOutput writeFloat(long offset, float d) {
+        throw new UnsupportedOperationException();
+    }
+
+
+    @Override
+    public RandomDataOutput writeDouble(long offset, double d) {
+        throw new UnsupportedOperationException();
+    }
+
+
+    @Override
+    public RandomDataOutput writeVolatileByte(long offset, byte i8) {
+        throw new UnsupportedOperationException();
+    }
+
+
+    @Override
+    public RandomDataOutput writeVolatileShort(long offset, short i16) {
+        throw new UnsupportedOperationException();
+    }
+
+
+    @Override
+    public RandomDataOutput writeVolatileInt(long offset, int i32) {
+        throw new UnsupportedOperationException();
+    }
+
+
+    @Override
+    public RandomDataOutput writeVolatileLong(long offset, long i64) {
+        throw new UnsupportedOperationException();
+    }
+
+
+    @Override
+    public RandomDataOutput write(long offsetInRDO, byte[] bytes, int offset, int length) {
+        if (length != 0)
+            throw new UnsupportedOperationException();
+        return this;
+    }
+
+    @Override
+    public void write(long offsetInRDO, ByteBuffer bytes, int offset, int length) {
+        if (length != 0)
+            throw new UnsupportedOperationException();
+    }
+
+
+    @Override
+    public RandomDataOutput write(long writeOffset, RandomDataInput bytes, long readOffset, long length) {
+        if (length != 0)
+            throw new UnsupportedOperationException();
+        return this;
+    }
+
+    @Override
+    public byte readByte(long offset) {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public int peekUnsignedByte(long offset) {
+        return -1;
+    }
+
+    @Override
+    public short readShort(long offset) {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public int readInt(long offset) {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public long readLong(long offset) {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public float readFloat(long offset) {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public double readDouble(long offset) {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public byte readVolatileByte(long offset) throws BufferUnderflowException {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public short readVolatileShort(long offset) throws BufferUnderflowException {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public int readVolatileInt(long offset) throws BufferUnderflowException {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public long readVolatileLong(long offset) throws BufferUnderflowException {
+        throw new BufferUnderflowException();
+    }
+
+    @Override
+    public boolean isDirectMemory() {
+        return false;
+    }
+
+
+    @Override
+    public BytesStore copy() {
+        return this;
+    }
+
+    @Override
+    public long capacity() {
+        return 0;
+    }
+
+    @Override
+    public Void underlyingObject() {
+        return null;
+    }
+
+    @Override
+    public boolean inside(long offset) {
+        return false;
+    }
+
+    @Override
+    public boolean inside(long offset, long buffer) {
+        return false;
+    }
+
+    @Override
+    public long copyTo( BytesStore store) {
+        // nothing to copy.
+        return 0L;
+    }
+
+    @Override
+    public void nativeWrite(long address, long position, long size) {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public void nativeRead(long position, long address, long size) {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public boolean compareAndSwapInt(long offset, int expected, int value) {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public void testAndSetInt(long offset, int expected, int value) {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public boolean compareAndSwapLong(long offset, long expected, long value) {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public boolean equalBytes( BytesStore bytesStore, long length) {
+        return length == 0;
+    }
+
+    @Override
+    public void move(long from, long to, long length) {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public long addressForRead(long offset) throws BufferUnderflowException {
+        if (offset != 0)
+            throw new BufferUnderflowException();
+        return NO_PAGE;
+    }
+
+    @Override
+    public long addressForWrite(long offset) throws BufferOverflowException {
+        if (offset != 0)
+            throw new BufferOverflowException();
+        return NO_PAGE;
+    }
+
+    @Override
+    public long addressForWritePosition() throws UnsupportedOperationException, BufferOverflowException {
+        return NO_PAGE;
+    }
+
+
+    @Override
+    public Bytes bytesForWrite() {
+        throw new UnsupportedOperationException("todo");
+    }
+
+    @Override
+    public boolean sharedMemory() {
+        return false;
+    }
+
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/OffsetFormat.java

@@ -0,0 +1,24 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+@FunctionalInterface
+public interface OffsetFormat {
+    @SuppressWarnings("rawtypes")
+    void append(long offset, Bytes bytes);
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/OnHeapBytes.java

@@ -0,0 +1,103 @@
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.bytes.util.DecoratedBufferOverflowException;
+import net.openhft.chronicle.core.Jvm;
+import org.jetbrains.annotations.NotNull;
+
+import java.nio.BufferOverflowException;
+
+public class OnHeapBytes extends VanillaBytes<byte[]> {
+    public static final int MAX_CAPACITY = Bytes.MAX_HEAP_CAPACITY;
+    private final boolean elastic;
+    private final long capacity;
+
+    public OnHeapBytes( BytesStore bytesStore, boolean elastic) throws IllegalStateException {
+        super(bytesStore);
+        this.elastic = elastic;
+        this.capacity = elastic ? MAX_CAPACITY : bytesStore.capacity();
+
+        writePosition(0);
+        writeLimit(capacity());
+    }
+
+    @Override
+    public long capacity() {
+        return capacity;
+    }
+
+    @Override
+    public boolean isElastic() {
+        return elastic;
+    }
+
+    @Override
+    protected void writeCheckOffset(long offset, long adding)
+            throws BufferOverflowException {
+        if (offset >= bytesStore.start()) {
+            long writeEnd = offset + adding;
+            if (writeEnd > writeLimit)
+                throwBeyondWriteLimit(adding, writeEnd);
+            if (writeEnd <= bytesStore.safeLimit()) {
+                return; // do nothing.
+            }
+            checkResize(writeEnd);
+        } else {
+            throw new BufferOverflowException();
+        }
+    }
+
+    private void throwBeyondWriteLimit(long advance, long writeEnd) {
+        throw new DecoratedBufferOverflowException("attempt to write " + advance + " bytes to " + writeEnd + " limit: " + writeLimit);
+    }
+
+    private void checkResize(long endOfBuffer)
+            throws BufferOverflowException {
+        if (isElastic())
+            resize(endOfBuffer);
+        else
+            throw new BufferOverflowException();
+    }
+
+    // the endOfBuffer is the minimum capacity and one byte more than the last addressable byte.
+    private void resize(long endOfBuffer)
+            throws BufferOverflowException {
+        if (endOfBuffer < 0)
+            throw new BufferOverflowException();
+        if (endOfBuffer > capacity())
+            throw new BufferOverflowException();
+        final long realCapacity = realCapacity();
+        if (endOfBuffer <= realCapacity) {
+//            System.out.println("no resize " + endOfBuffer + " < " + realCapacity);
+            return;
+        }
+
+        // Grow by 50%
+        long size0 = Math.max(endOfBuffer, realCapacity * 3 / 2);
+        // Size must not be more than capacity(), it may break some assumptions in BytesStore or elsewhere
+        int size = (int) Math.min(size0, capacity());
+
+        //        System.out.println("resize " + endOfBuffer + " to " + size);
+        if (endOfBuffer > 1 << 20)
+            Jvm.warn().on(getClass(), "Resizing buffer was " + realCapacity / 1024 + " KB, " +
+                    "needs " + (endOfBuffer - realCapacity) + " bytes more, " +
+                    "new-size " + size / 1024 + " KB");
+        HeapBytesStore store;
+        try {
+            store = HeapBytesStore.wrap(new byte[size]);
+            store.reserveTransfer(INIT, this);
+        } catch (IllegalArgumentException e) {
+            BufferOverflowException boe = new BufferOverflowException();
+            boe.initCause(e);
+            throw boe;
+        }
+
+        BytesStore<Bytes<byte[]>, byte[]> tempStore = this.bytesStore;
+        this.bytesStore.copyTo(store);
+        this.bytesStore(store);
+        try {
+            tempStore.release(this);
+        } catch (IllegalStateException e) {
+            Jvm.debug().on(getClass(), e);
+        }
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/PointerBytesStore.java

@@ -0,0 +1,55 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * A BytesStore which can point to arbitrary memory.
+ */
+public class PointerBytesStore extends NativeBytesStore<Void> {
+    private boolean isPresent = false;
+
+    public PointerBytesStore() {
+        super(NoBytesStore.NO_PAGE, 0, null, false, false);
+    }
+
+    public void set(long address, long capacity) {
+        isPresent = true;
+        setAddress(address);
+        this.limit = maximumLimit = capacity;
+    }
+
+
+    @Override
+    public VanillaBytes<Void> bytesForWrite() throws IllegalStateException {
+        return new VanillaBytes<>(this);
+    }
+
+    @Override
+    public long safeLimit() {
+        return limit;
+    }
+
+    @Override
+    public long start() {
+        return 0;
+    }
+    
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/RandomCommon.java

@@ -0,0 +1,207 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.annotation.ForceInline;
+import net.openhft.chronicle.core.io.ReferenceCounted;
+import org.jetbrains.annotations.NotNull;
+
+import java.nio.BufferOverflowException;
+import java.nio.BufferUnderflowException;
+import java.nio.ByteOrder;
+
+interface RandomCommon extends ReferenceCounted {
+    /**
+     * @return The smallest position allowed in this buffer.
+     */
+    @ForceInline
+    default long start() {
+        return 0L;
+    }
+
+    /**
+     * @return the highest limit allowed for this buffer.
+     */
+    @ForceInline
+    default long capacity() {
+        return Bytes.MAX_CAPACITY;
+    }
+
+    /**
+     * @return the limit for this buffer without resizing
+     */
+    default long realCapacity() {
+        return capacity();
+    }
+
+    /**
+     * The read position must be start() <= readPosition() && readPosition() <= readLimit() && readPosition < safeLimit()
+     *
+     * @return position to read from.
+     */
+    @ForceInline
+    default long readPosition() {
+        return start();
+    }
+
+    /**
+     * The read position must be readPosition() <= writePosition() && writePosition() <= writeLimit()
+     *
+     * @return position to write to.
+     */
+    @ForceInline
+    default long writePosition() {
+        return start();
+    }
+
+    /**
+     * @return How many more bytes can we read.
+     */
+    @ForceInline
+    default long readRemaining() {
+        long remaining = readLimit() - readPosition();
+        return remaining;
+    }
+
+    /**
+     * @return How many more bytes can we written.
+     */
+    @ForceInline
+    default long writeRemaining() {
+        return writeLimit() - writePosition();
+    }
+
+    /**
+     * @return the highest offset or position allowed for this buffer.
+     */
+    @ForceInline
+    default long readLimit() {
+        return realCapacity();
+    }
+
+    @ForceInline
+    default long writeLimit() {
+        return realCapacity();
+    }
+
+    /**
+     * Obtain the underlying addressForRead.  This is for expert users only.
+     *
+     * @param offset within this buffer. addressForRead(start()) is the actual addressForRead of the first byte.
+     * @return the underlying addressForRead of the buffer
+     * @throws UnsupportedOperationException if the underlying buffer is on the heap
+     * @throws BufferUnderflowException      if the offset is before the start() or the after the capacity()
+     */
+    long addressForRead(long offset)
+            throws UnsupportedOperationException, BufferUnderflowException;
+
+    default long addressForRead(long offset, int buffer)
+            throws UnsupportedOperationException, BufferUnderflowException {
+        return addressForRead(offset);
+    }
+
+    /**
+     * Obtain the underlying addressForRead.  This is for expert users only.
+     *
+     * @param offset within this buffer. addressForRead(start()) is the actual addressForRead of the first byte.
+     * @return the underlying addressForRead of the buffer
+     * @throws UnsupportedOperationException if the underlying buffer is on the heap
+     * @throws BufferOverflowException       if the offset is before the start() or the after the capacity()
+     */
+    long addressForWrite(long offset)
+            throws UnsupportedOperationException, BufferOverflowException;
+
+    long addressForWritePosition()
+            throws UnsupportedOperationException, BufferOverflowException;
+
+    default ByteOrder byteOrder() {
+        return ByteOrder.nativeOrder();
+    }
+
+    /**
+     * @return the streaming bytes for reading.
+     */
+    @SuppressWarnings("rawtypes")
+
+    Bytes bytesForRead() throws IllegalStateException;
+
+    /**
+     * @return the streaming bytes for writing.
+     */
+    @SuppressWarnings("rawtypes")
+
+    Bytes bytesForWrite() throws IllegalStateException;
+
+    /**
+     * Perform a 32-bit CAS at a given offset.
+     *
+     * @param offset   to perform CAS
+     * @param expected value
+     * @param value    to set
+     * @return true, if successful.
+     */
+    boolean compareAndSwapInt(long offset, int expected, int value)
+            throws BufferOverflowException;
+
+    void testAndSetInt(long offset, int expected, int value);
+
+    /**
+     * Perform a 64-bit CAS at a given offset.
+     *
+     * @param offset   to perform CAS
+     * @param expected value
+     * @param value    to set
+     * @return true, if successful.
+     */
+    boolean compareAndSwapLong(long offset, long expected, long value)
+            throws BufferOverflowException;
+
+    /**
+     * Perform a 32-bit float CAS at a given offset.
+     *
+     * @param offset   to perform CAS
+     * @param expected value
+     * @param value    to set
+     * @return true, if successful.
+     */
+    default boolean compareAndSwapFloat(long offset, float expected, float value)
+            throws BufferOverflowException {
+        return compareAndSwapInt(offset, Float.floatToRawIntBits(expected), Float.floatToRawIntBits(value));
+    }
+
+    /**
+     * Perform a 64-bit double CAS at a given offset.
+     *
+     * @param offset   to perform CAS
+     * @param expected value
+     * @param value    to set
+     * @return true, if successful.
+     */
+    default boolean compareAndSwapDouble(long offset, double expected, double value)
+            throws BufferOverflowException {
+        return compareAndSwapLong(offset, Double.doubleToRawLongBits(expected), Double.doubleToRawLongBits(value));
+    }
+
+    /**
+     * @return true if these Bytes use shared memory.
+     */
+    boolean sharedMemory();
+
+    boolean isDirectMemory();
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/RandomDataInput.java

@@ -0,0 +1,578 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.Maths;
+import net.openhft.chronicle.core.annotation.ForceInline;
+import net.openhft.chronicle.core.io.IORuntimeException;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+
+/**
+ * This allows random access to the underling bytes.  This instance can be used across threads as it is stateless.
+ * The thread safety of the underlying data depends on how the methods are used.
+ */
+public interface RandomDataInput extends RandomCommon {
+    String[] charToString = createCharToString();
+
+
+    static String[] createCharToString() {
+         String[] charToString = new String[256];
+        charToString[0] = "\u0660";
+        for (int i = 1; i < 21; i++)
+            charToString[i] = Character.toString((char) (i + 0x2487));
+        for (int i = ' '; i < 256; i++)
+            charToString[i] = Character.toString((char) i);
+        for (int i = 21; i < ' '; i++)
+            charToString[i] = "\\u00" + Integer.toHexString(i).toUpperCase();
+        for (int i = 0x80; i < 0xA0; i++)
+            charToString[i] = "\\u00" + Integer.toHexString(i).toUpperCase();
+        return charToString;
+    }
+
+    default int peekVolatileInt() throws BufferUnderflowException {
+        return readVolatileInt(readPosition());
+    }
+
+    /**
+     * Read boolean at an offset
+     *
+     * @param offset to read
+     * @return the boolean
+     * @throws BufferUnderflowException if the offset is outside the limits of the Bytes
+     */
+    @ForceInline
+    default boolean readBoolean(long offset)
+            throws BufferUnderflowException {
+        return BytesUtil.byteToBoolean(readByte(offset));
+    }
+
+    /**
+     * Read byte at an offset
+     *
+     * @param offset to read
+     * @return the byte
+     * @throws BufferUnderflowException if the offset is outside the limits of the Bytes
+     */
+    byte readByte(long offset) throws BufferUnderflowException;
+
+    /**
+     * Read an unsigned byte at an offset
+     *
+     * @param offset to read
+     * @return the unsigned byte
+     * @throws BufferUnderflowException if the offset is outside the limits of the Bytes
+     */
+    @ForceInline
+    default int readUnsignedByte(long offset) throws BufferUnderflowException {
+        return readByte(offset) & 0xFF;
+    }
+
+    /**
+     * Read an unsigned byte at an offset, or -1
+     *
+     * @param offset to read
+     * @return the unsigned byte or -1
+     */
+    int peekUnsignedByte(long offset) throws BufferUnderflowException;
+
+    /**
+     * Read a short at an offset
+     *
+     * @param offset to read
+     * @return the short
+     * @throws BufferUnderflowException if the offset is outside the limits of the Bytes
+     */
+    short readShort(long offset) throws BufferUnderflowException;
+
+    /**
+     * Read an unsigned short at an offset
+     *
+     * @param offset to read
+     * @return the unsigned short
+     * @throws BufferUnderflowException if the offset is outside the limits of the Bytes
+     */
+    @ForceInline
+    default int readUnsignedShort(long offset) throws BufferUnderflowException {
+        return readShort(offset) & 0xFFFF;
+    }
+
+    /**
+     * Read an unsigned int at an offset
+     *
+     * @param offset to read
+     * @return the int
+     * @throws BufferUnderflowException if the offset is outside the limits of the Bytes
+     */
+    default int readUnsignedInt24(long offset) throws BufferUnderflowException {
+        return readUnsignedShort(offset) | (readUnsignedByte(offset) << 16);
+    }
+
+    /**
+     * Read an int at an offset
+     *
+     * @param offset to read
+     * @return the int
+     * @throws BufferUnderflowException if the offset is outside the limits of the Bytes
+     */
+    int readInt(long offset) throws BufferUnderflowException;
+
+    /**
+     * Read an unsigned int at an offset
+     *
+     * @param offset to read
+     * @return the unsigned int
+     * @throws BufferUnderflowException if the offset is outside the limits of the Bytes
+     */
+    @ForceInline
+    default long readUnsignedInt(long offset) throws BufferUnderflowException {
+        return readInt(offset) & 0xFFFFFFFFL;
+    }
+
+    /**
+     * Read a long at an offset
+     *
+     * @param offset to read
+     * @return the long
+     * @throws BufferUnderflowException if the offset is outside the limits of the Bytes
+     */
+    long readLong(long offset) throws BufferUnderflowException;
+
+    /**
+     * Read a float at an offset
+     *
+     * @param offset to read
+     * @return the float
+     * @throws BufferUnderflowException if the offset is outside the limits of the Bytes
+     */
+    float readFloat(long offset) throws BufferUnderflowException;
+
+    /**
+     * Read a double at an offset
+     *
+     * @param offset to read
+     * @return the double
+     * @throws BufferUnderflowException if the offset is outside the limits of the Bytes
+     */
+    double readDouble(long offset) throws BufferUnderflowException;
+
+    /**
+     * Read the byte at an offset and converts it into a printable
+     *
+     * @param offset to read
+     * @return the byte in a printable form.
+     * @throws BufferUnderflowException if the offset is outside the limits of the Bytes
+     */
+    default String printable(long offset) throws BufferUnderflowException {
+        return charToString[readUnsignedByte(offset)];
+    }
+
+    /**
+     * Read a 8-bit byte from memory with a load barrier.
+     *
+     * @param offset to read
+     * @return the byte value
+     * @throws BufferUnderflowException if the offset is outside the limits of the Bytes
+     */
+    byte readVolatileByte(long offset) throws BufferUnderflowException;
+
+    /**
+     * Read a 16-bit short from memory with a load barrier.
+     *
+     * @param offset to read
+     * @return the short value
+     * @throws BufferUnderflowException if the offset is outside the limits of the Bytes
+     */
+    short readVolatileShort(long offset) throws BufferUnderflowException;
+
+    /**
+     * Read a 32-bit int from memory with a load barrier.
+     *
+     * @param offset to read
+     * @return the int value
+     * @throws BufferUnderflowException if the offset is outside the limits of the Bytes
+     */
+    int readVolatileInt(long offset) throws BufferUnderflowException;
+
+    /**
+     * Read a float from memory with a load barrier.
+     *
+     * @param offset to read
+     * @return the float value
+     * @throws BufferUnderflowException if the offset is outside the limits of the Bytes
+     */
+    default float readVolatileFloat(long offset) throws BufferUnderflowException {
+        return Float.intBitsToFloat(readVolatileInt(offset));
+    }
+
+    /**
+     * Read a 64-bit long from memory with a load barrier.
+     *
+     * @param offset to read
+     * @return the long value
+     * @throws BufferUnderflowException if the offset is outside the limits of the Bytes
+     */
+    long readVolatileLong(long offset) throws BufferUnderflowException;
+
+    /**
+     * Read a 64-bit double from memory with a load barrier.
+     *
+     * @param offset to read
+     * @return the double value
+     * @throws BufferUnderflowException if the offset is outside the limits of the Bytes
+     */
+    default double readVolatileDouble(long offset) throws BufferUnderflowException {
+        return Double.longBitsToDouble(readVolatileLong(offset));
+    }
+
+    default long parseLong(long offset) throws BufferUnderflowException {
+        return BytesInternal.parseLong(this, offset);
+    }
+
+    /**
+     * expert level method for copying data to native memory.
+     *
+     * @param position within the ByteStore to copy.
+     * @param address  in native memory
+     * @param size     in bytes
+     * @throws BufferUnderflowException if the offset is outside the limits of the Bytes
+     */
+    void nativeRead(long position, long address, long size) throws BufferUnderflowException;
+
+    /**
+     * Read a byte[] from memory.
+     *
+     * @return the length actually read.
+     * @throws BufferUnderflowException if the offset is outside the limits of the Bytes
+     */
+    default int copyTo( byte[] bytes) throws BufferUnderflowException {
+        int len = (int) Math.min(bytes.length, readRemaining());
+        for (int i = 0; i < len; i++)
+            bytes[i] = readByte(start() + i);
+        return len;
+    }
+
+    /**
+     * Copy data from this RandomDataInput to the ByteBuffer. The minimum of {@link #readRemaining()} and
+     * {@link ByteBuffer#remaining()}. Starting from {@link #start()} in this RandomDataInput and from {@link
+     * ByteBuffer#position()} of the given bb. Does NOT change the position or limit or mark of the given ByteBuffer.
+     * Returns the number of the copied bytes.
+     */
+    default int copyTo( ByteBuffer bb) throws BufferUnderflowException {
+        int pos = bb.position();
+        int len = (int) Math.min(bb.remaining(), readRemaining());
+        int i;
+        for (i = 0; i < len - 7; i += 8)
+            bb.putLong(pos + i, readLong(start() + i));
+        for (; i < len; i++)
+            bb.put(pos + i, readByte(start() + i));
+        return len;
+    }
+
+    /**
+     * Read a long which is zero padded (high bytes) if the available bytes is less than 8.
+     * If the offset is at or beyond the readLimit, this will return 0L.
+     *
+     * @param offset to read from
+     * @return the long which might be padded.
+     */
+    default long readIncompleteLong(long offset) {
+        long left = readLimit() - offset;
+        long l;
+        try {
+            if (left >= 8)
+                return readLong(offset);
+            if (left == 4)
+                return readInt(offset);
+            l = 0;
+            for (int i = 0, remaining = (int) left; i < remaining; i++) {
+                l |= (long) readUnsignedByte(offset + i) << (i * 8);
+            }
+        } catch (BufferUnderflowException e) {
+            throw new AssertionError(e);
+        }
+        return l;
+    }
+
+    /**
+     * Returns the actual capacity that can be potentially read.
+     *
+     * @return the actual capacity that can be potentially read.
+     */
+    long realCapacity();
+
+    /**
+     * Perform an atomic add and get operation for a 32-bit int
+     *
+     * @param offset to add and get
+     * @param adding value to add, can be 1
+     * @return the sum
+     * @throws BufferUnderflowException if the offset is outside the limits of the Bytes
+     */
+    default int addAndGetInt(long offset, int adding)
+            throws BufferUnderflowException {
+        return BytesInternal.addAndGetInt(this, offset, adding);
+    }
+
+    /**
+     * Perform an atomic add and get operation for a 64-bit long
+     *
+     * @param offset to add and get
+     * @param adding value to add, can be 1
+     * @return the sum
+     * @throws BufferUnderflowException if the offset is outside the limits of the Bytes
+     */
+    default long addAndGetLong(long offset, long adding)
+            throws BufferUnderflowException {
+        return BytesInternal.addAndGetLong(this, offset, adding);
+    }
+
+    /**
+     * Perform an atomic add and get operation for a 32-bit float
+     *
+     * @param offset to add and get
+     * @param adding value to add, can be 1
+     * @return the sum
+     * @throws BufferUnderflowException if the offset is outside the limits of the Bytes
+     */
+    default float addAndGetFloat(long offset, float adding)
+            throws BufferUnderflowException {
+        return BytesInternal.addAndGetFloat(this, offset, adding);
+    }
+
+    /**
+     * Perform an atomic add and get operation for a 64-bit double
+     *
+     * @param offset to add and get
+     * @param adding value to add, can be 1
+     * @return the sum
+     * @throws BufferUnderflowException if the offset is outside the limits of the Bytes
+     */
+    default double addAndGetDouble(long offset, double adding)
+            throws BufferUnderflowException {
+        return BytesInternal.addAndGetDouble(this, offset, adding);
+    }
+
+    /**
+     * Copy a sub sequence of bytes as a BytesStore.
+     *
+     * @param start  of bytes
+     * @param length of bytes
+     * @return ByteStore copy.
+     */
+    @SuppressWarnings("rawtypes")
+
+    default BytesStore subBytes(long start, long length) throws BufferUnderflowException {
+        return BytesInternal.subBytes(this, start, length);
+    }
+
+    default long findByte(byte stopByte) {
+        return BytesInternal.findByte(this, stopByte);
+    }
+
+    /**
+     * Truncates {@code sb} (it must be a {@link StringBuilder} or {@link Bytes}) and reads a char
+     * sequence from the given {@code offset}, encoded as Utf8, into it. Returns offset <i>after</i>
+     * the read Utf8, if a normal char sequence was read, or {@code -1 - offset}, if {@code null}
+     * was observed (in this case, {@code sb} is truncated too, but not updated then, by querying
+     * {@code sb} only this case is indistinguishable from reading an empty char sequence).
+     *
+     * @param offset the offset in this {@code RandomDataInput} to read char sequence from
+     * @param sb     the buffer to read char sequence into (truncated first)
+     * @param <ACS>  buffer type, must be {@code StringBuilder} or {@code Bytes}
+     * @return offset after the normal read char sequence, or -1 - offset, if char sequence is
+     * {@code null}
+     * @see RandomDataOutput#writeUtf8(long, CharSequence)
+     */
+    default <ACS extends Appendable & CharSequence> long readUtf8(long offset,  ACS sb)
+            throws IORuntimeException, IllegalArgumentException, BufferUnderflowException {
+        AppendableUtil.setLength(sb, 0);
+        // TODO insert some bounds check here
+
+        long utfLen;
+        if ((utfLen = readByte(offset++)) < 0) {
+            utfLen &= 0x7FL;
+            long b;
+            int count = 7;
+            while ((b = readByte(offset++)) < 0) {
+                utfLen |= (b & 0x7FL) << count;
+                count += 7;
+            }
+            if (b != 0) {
+                if (count > 56)
+                    throw new IORuntimeException(
+                            "Cannot read more than 9 stop bits of positive value");
+                utfLen |= (b << count);
+            } else {
+                if (count > 63)
+                    throw new IORuntimeException(
+                            "Cannot read more than 10 stop bits of negative value");
+                utfLen = ~utfLen;
+            }
+        }
+        if (utfLen == -1)
+            return ~offset;
+        int len = Maths.toUInt31(utfLen);
+        BytesInternal.parseUtf8(this, offset, sb, true, len);
+        return offset + utfLen;
+    }
+
+    /**
+     * Truncates {@code sb} (it must be a {@link StringBuilder} or {@link Bytes}) and reads a char
+     * sequence from the given {@code offset}, encoded as Utf8, into it. Returns offset <i>after</i>
+     * the read Utf8, if a normal char sequence was read, or {@code -1 - offset}, if {@code null}
+     * was observed (in this case, {@code sb} is truncated too, but not updated then, by querying
+     * {@code sb} only this case is indistinguishable from reading an empty char sequence). If
+     * length of Utf8 encoding of the char sequence exceeds {@code maxUtf8Len},
+     * {@code IllegalStateException} is thrown.
+     *
+     * @param offset     the offset in this {@code RandomDataInput} to read char sequence from
+     * @param sb         the buffer to read char sequence into (truncated first)
+     * @param maxUtf8Len the maximum allowed length of the char sequence in Utf8 encoding
+     * @param <ACS>      buffer type, must be {@code StringBuilder} or {@code Bytes}
+     * @return offset after the normal read char sequence, or -1 - offset, if char sequence is
+     * {@code null}
+     * @see RandomDataOutput#writeUtf8Limited(long, CharSequence, int)
+     */
+    default <ACS extends Appendable & CharSequence> long readUtf8Limited(
+            long offset,  ACS sb, int maxUtf8Len)
+            throws IORuntimeException, IllegalArgumentException, BufferUnderflowException,
+            IllegalStateException {
+        AppendableUtil.setLength(sb, 0);
+        // TODO insert some bounds check here
+
+        long utfLen;
+        if ((utfLen = readByte(offset++)) < 0) {
+            utfLen &= 0x7FL;
+            long b;
+            int count = 7;
+            while ((b = readByte(offset++)) < 0) {
+                utfLen |= (b & 0x7FL) << count;
+                count += 7;
+            }
+            if (b != 0) {
+                if (count > 56)
+                    throw new IORuntimeException(
+                            "Cannot read more than 9 stop bits of positive value");
+                utfLen |= (b << count);
+            } else {
+                if (count > 63)
+                    throw new IORuntimeException(
+                            "Cannot read more than 10 stop bits of negative value");
+                utfLen = ~utfLen;
+            }
+        }
+        if (utfLen == -1)
+            return ~offset;
+        if (utfLen > maxUtf8Len)
+            throw new IllegalStateException("Attempted to read a char sequence of " +
+                    "utf8 size " + utfLen + ", when only " + maxUtf8Len + " allowed");
+        BytesInternal.parseUtf8(this, offset, sb, true, (int) utfLen);
+        return offset + utfLen;
+    }
+
+    /**
+     * Reads a char sequence from the given {@code offset}, encoded as Utf8. If length of Utf8
+     * encoding of the char sequence exceeds {@code maxUtf8Len}, {@code IllegalStateException}
+     * is thrown.
+     *
+     * @param offset     the offset in this {@code RandomDataInput} to read char sequence from
+     * @param maxUtf8Len the maximum allowed length of the char sequence in Utf8 encoding
+     * @return the char sequence was read
+     * @see RandomDataOutput#writeUtf8Limited(long, CharSequence, int)
+     */
+
+    default String readUtf8Limited(long offset, int maxUtf8Len)
+            throws BufferUnderflowException, IORuntimeException, IllegalArgumentException,
+            IllegalStateException {
+        return BytesInternal.readUtf8(this, offset, maxUtf8Len);
+    }
+
+    /**
+     * Compares the UTF-8 encoded char sequence, written in this {@code RandomDataInput} at the
+     * given offset, with the given char sequence. Returns {@code true}, if they are equal. Both
+     * char sequences (encoded in bytes and the given) may be {@code null}.
+     *
+     * @param offset the offset in this {@code RandomDataInput} where the char sequence to compare
+     *               is written
+     * @param other  the second char sequence to compare
+     * @return {@code true} if two char sequences are equal
+     * @throws IORuntimeException if the contents are not a valid string.
+     */
+    default boolean compareUtf8(long offset,  CharSequence other) throws IORuntimeException {
+        return BytesInternal.compareUtf8(this, offset, other);
+    }
+
+
+    default byte[] toByteArray() throws IllegalArgumentException {
+        return BytesInternal.toByteArray(this);
+    }
+
+    default long read(long offsetInRDI, byte[] bytes, int offset, int length) {
+        try {
+            int len = (int) Math.min(length, readLimit() - offsetInRDI);
+            for (int i = 0; i < len; i++)
+                bytes[offset + i] = readByte(offsetInRDI + i);
+            return len;
+        } catch (BufferUnderflowException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+    default ByteBuffer toTemporaryDirectByteBuffer() throws IllegalArgumentException {
+        int len = Maths.toUInt31(readRemaining());
+        try {
+            ByteBuffer bb = ByteBuffer.allocateDirect(len);
+            copyTo(bb);
+            bb.clear();
+            return bb;
+        } catch (BufferUnderflowException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+    default int fastHash(long offset, int length) throws BufferUnderflowException {
+        long hash = 0;
+        int i = 0;
+        if (length >= 4) {
+            hash = readInt(offset + i);
+            i += 4;
+        }
+        for (; i < length - 3; i += 4) {
+            hash *= 0x6d0f27bd;
+            hash += readInt(offset + i);
+        }
+        if (i < length - 1) {
+            hash *= 0x6d0f27bdL;
+            hash += readShort(offset + i);
+            i += 2;
+        }
+        if (i < length)
+            hash += readByte(offset + i);
+        hash *= 0x855dd4db;
+        return (int) (hash ^ (hash >> 32));
+    }
+
+    default boolean canReadDirect(long length) {
+        return isDirectMemory() && readRemaining() >= length;
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/RandomDataOutput.java

@@ -0,0 +1,351 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.Maths;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+import java.nio.BufferOverflowException;
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+
+@SuppressWarnings({"rawtypes", "unchecked"})
+public interface RandomDataOutput<R extends RandomDataOutput<R>> extends RandomCommon {
+    /**
+     * Write a byte at an offset.
+     *
+     * @param offset to write to
+     * @param i      the value
+     * @return this
+     * @throws BufferOverflowException  if the capacity was exceeded
+     * @throws IllegalArgumentException if the value cannot be cast to the type without loss.
+     */
+
+    default R writeByte(long offset, int i)
+            throws BufferOverflowException, IllegalArgumentException {
+        return writeByte(offset, Maths.toInt8(i));
+    }
+
+    /**
+     * Write an unsigned byte at an offset.
+     *
+     * @param offset to write to
+     * @param i      the value
+     * @return this
+     * @throws BufferOverflowException  if the capacity was exceeded
+     * @throws IllegalArgumentException if the value cannot be cast to the type without loss.
+     */
+
+    default R writeUnsignedByte(long offset, int i)
+            throws BufferOverflowException, IllegalArgumentException {
+        return writeByte(offset, (byte) Maths.toUInt8(i));
+    }
+
+    /**
+     * Write a boolean at an offset.
+     *
+     * @param offset to write to
+     * @param flag   the value
+     * @return this
+     * @throws BufferOverflowException if the capacity was exceeded
+     */
+
+    default R writeBoolean(long offset, boolean flag)
+            throws BufferOverflowException {
+        try {
+            return writeByte(offset, flag ? 'Y' : 'N');
+
+        } catch (IllegalArgumentException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+    /**
+     * Write an unsigned byte at an offset.
+     *
+     * @param offset to write to
+     * @param i      the value
+     * @return this
+     * @throws BufferOverflowException  if the capacity was exceeded
+     * @throws IllegalArgumentException if the value cannot be cast to the type without loss.
+     */
+
+    default R writeUnsignedShort(long offset, int i)
+            throws BufferOverflowException, IllegalArgumentException {
+        return writeShort(offset, (short) Maths.toUInt16(i));
+    }
+
+    /**
+     * Write an unsigned byte at an offset.
+     *
+     * @param offset to write to
+     * @param i      the value
+     * @return this
+     * @throws BufferOverflowException  if the capacity was exceeded
+     * @throws IllegalArgumentException if the value cannot be cast to the type without loss.
+     */
+
+    default R writeUnsignedInt(long offset, long i)
+            throws BufferOverflowException, IllegalArgumentException {
+        return writeInt(offset, (int) Maths.toUInt32(i));
+    }
+
+    /**
+     * Write an unsigned byte at an offset.
+     *
+     * @param offset to write to
+     * @param i8     the value
+     * @return this
+     * @throws BufferOverflowException if the capacity was exceeded
+     */
+
+    R writeByte(long offset, byte i8) throws BufferOverflowException;
+
+    /**
+     * Write a short at an offset.
+     *
+     * @param offset to write to
+     * @param i      the value
+     * @return this
+     * @throws BufferOverflowException if the capacity was exceeded
+     */
+
+    R writeShort(long offset, short i) throws BufferOverflowException;
+
+
+    default R writeInt24(long offset, int i) throws BufferOverflowException {
+        writeShort(offset, (short) i);
+        return writeByte(offset + 2, (byte) (i >> 16));
+    }
+
+    /**
+     * Write an int at an offset.
+     *
+     * @param offset to write to
+     * @param i      the value
+     * @return this
+     * @throws BufferOverflowException if the capacity was exceeded
+     */
+
+    R writeInt(long offset, int i) throws BufferOverflowException;
+
+    /**
+     * Perform a non stalling write with a store barrier.
+     *
+     * @param offset to write to
+     * @param i      value to write
+     * @return this
+     * @throws BufferOverflowException if the capacity was exceeded
+     */
+
+    R writeOrderedInt(long offset, int i) throws BufferOverflowException;
+
+    /**
+     * Perform a non stalling write with a store barrier.
+     *
+     * @param offset to write to
+     * @param f      value to write
+     * @return this
+     * @throws BufferOverflowException if the capacity was exceeded
+     */
+
+    default R writeOrderedFloat(long offset, float f) throws BufferOverflowException {
+        return writeOrderedInt(offset, Float.floatToRawIntBits(f));
+    }
+
+    /**
+     * Write a long at an offset.
+     *
+     * @param offset to write to
+     * @param i      the value
+     * @return this
+     * @throws BufferOverflowException if the capacity was exceeded
+     */
+
+    R writeLong(long offset, long i) throws BufferOverflowException, IllegalStateException;
+
+    /**
+     * Perform a non stalling write with a store barrier.
+     *
+     * @param offset to write to
+     * @param i      value to write
+     * @return this
+     */
+
+    R writeOrderedLong(long offset, long i) throws BufferOverflowException;
+
+    /**
+     * Perform a non stalling write with a store barrier.
+     *
+     * @param offset to write to
+     * @param d      value to write
+     * @return this
+     */
+
+    default R writeOrderedDouble(long offset, double d) throws BufferOverflowException {
+        return writeOrderedLong(offset, Double.doubleToRawLongBits(d));
+    }
+
+    /**
+     * Write a float at an offset.
+     *
+     * @param offset to write to
+     * @param d      the value
+     * @return this
+     * @throws BufferOverflowException if the capacity was exceeded
+     */
+
+    R writeFloat(long offset, float d) throws BufferOverflowException;
+
+    /**
+     * Write a double at an offset.
+     *
+     * @param offset to write to
+     * @param d      the value
+     * @return this
+     * @throws BufferOverflowException if the capacity was exceeded
+     */
+
+    R writeDouble(long offset, double d) throws BufferOverflowException;
+
+
+    R writeVolatileByte(long offset, byte i8) throws BufferOverflowException;
+
+
+    R writeVolatileShort(long offset, short i16) throws BufferOverflowException;
+
+
+    R writeVolatileInt(long offset, int i32) throws BufferOverflowException;
+
+
+    R writeVolatileLong(long offset, long i64) throws BufferOverflowException;
+
+
+    default R writeVolatileFloat(long offset, float f) throws BufferOverflowException {
+        return writeVolatileInt(offset, Float.floatToRawIntBits(f));
+    }
+
+
+    default R writeVolatileDouble(long offset, double d) throws BufferOverflowException {
+        return writeVolatileLong(offset, Double.doubleToRawLongBits(d));
+    }
+
+
+    default R write(long offsetInRDO,  byte[] bytes) throws BufferOverflowException {
+        return write(offsetInRDO, bytes, 0, bytes.length);
+    }
+
+
+    R write(long offsetInRDO, byte[] bytes, int offset, int length)
+            throws BufferOverflowException;
+
+    void write(long offsetInRDO, ByteBuffer bytes, int offset, int length)
+            throws BufferOverflowException;
+
+
+    default R write(long offsetInRDO,  BytesStore bytes)
+            throws BufferOverflowException {
+        try {
+            return write(offsetInRDO, bytes, bytes.readPosition(), bytes.readRemaining());
+
+        } catch (BufferUnderflowException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+
+    R write(long writeOffset, RandomDataInput bytes, long readOffset, long length)
+            throws BufferOverflowException, BufferUnderflowException;
+
+    /**
+     * Zero out the bytes between the start and the end.
+     *
+     * @param start index of first byte inclusive
+     * @param end   index of last byte exclusive
+     * @return this
+     * @throws BufferOverflowException if the capacity was exceeded
+     */
+
+    R zeroOut(long start, long end);
+
+
+    default R append(long offset, long value, int digits)
+            throws BufferOverflowException {
+        BytesInternal.append(this, offset, value, digits);
+        return (R) this;
+    }
+
+
+    default R append(long offset, double value, int decimalPlaces, int digits) throws BufferOverflowException {
+        if (decimalPlaces < 20) {
+            double d2 = value * Maths.tens(decimalPlaces);
+            if (d2 <= Long.MAX_VALUE && d2 >= Long.MIN_VALUE) {
+                BytesInternal.appendDecimal(this, Math.round(d2), offset, decimalPlaces, digits);
+                return (R) this;
+            }
+        }
+        BytesInternal.append((StreamingDataOutput) this, value);
+        return (R) this;
+    }
+
+    /**
+     * expert level method to copy data from native memory into the BytesStore
+     *
+     * @param address  in native memory to copy from
+     * @param position in BytesStore to copy to
+     * @param size     in bytes
+     */
+    void nativeWrite(long address, long position, long size) throws BufferOverflowException;
+
+    /**
+     * Writes the given {@code cs} to this {@code RandomDataOutput} from the given {@code offset},
+     * in Utf8 format. Returns the offset after the written char sequence.
+     *
+     * @param offset the offset to write char sequence from
+     * @param cs     the char sequence to write, could be {@code null}
+     * @return the offset after the char sequence written, in this {@code RandomDataOutput}
+     * @see RandomDataInput#readUtf8(long, Appendable)
+     */
+    default long writeUtf8(long offset,  CharSequence cs) throws BufferOverflowException {
+        return BytesInternal.writeUtf8(this, offset, cs);
+    }
+
+    /**
+     * Writes the given {@code cs} to this {@code RandomDataOutput} from the given {@code offset},
+     * in Utf8 format, checking that the utf8 encoding size of the given char sequence is less or
+     * equal to the given {@code maxUtf8Len}, otherwise {@code IllegalArgumentException} is thrown,
+     * and no bytes of this {@code RandomDataOutput} are overwritten. Returns the offset after the
+     * written char sequence.
+     *
+     * @param offset     the offset to write char sequence from
+     * @param cs         the char sequence to write, could be {@code null}
+     * @param maxUtf8Len the maximum allowed length (in Utf8 encoding) of the given char sequence
+     * @return the offset after the char sequence written, in this {@code RandomDataOutput}
+     * @throws IllegalArgumentException if the given char sequence size in Utf8 encoding exceeds
+     *                                  maxUtf8Len
+     * @see RandomDataInput#readUtf8Limited(long, Appendable, int)
+     * @see RandomDataInput#readUtf8Limited(long, int)
+     */
+    default long writeUtf8Limited(long offset,  CharSequence cs, int maxUtf8Len)
+            throws BufferOverflowException {
+        return BytesInternal.writeUtf8(this, offset, cs, maxUtf8Len);
+    }
+
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/ReadBytesMarshallable.java

@@ -0,0 +1,37 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.annotation.DontChain;
+import net.openhft.chronicle.core.io.IORuntimeException;
+
+/**
+ * Read data directly as Bytes.
+ */
+@FunctionalInterface
+@DontChain
+public interface ReadBytesMarshallable extends CommonMarshallable {
+    /**
+     * Bytes to read.  This can be used as a method to implement or as a lambda.
+     *
+     * @param bytes to read.
+     */
+    @SuppressWarnings("rawtypes")
+    void readMarshallable(BytesIn bytes) throws IORuntimeException;
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/ReadOnlyMappedBytesStore.java

@@ -0,0 +1,183 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.io.ReferenceOwner;
+import org.jetbrains.annotations.NotNull;
+
+import java.nio.BufferOverflowException;
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+
+/**
+ * BytesStore to wrap memory mapped data.
+ */
+public class ReadOnlyMappedBytesStore extends MappedBytesStore {
+
+    public ReadOnlyMappedBytesStore(ReferenceOwner owner, MappedFile mappedFile, long start, long address, long capacity, long safeCapacity)
+            throws IllegalStateException {
+        super(owner, mappedFile, start, address, capacity, safeCapacity);
+    }
+
+
+    @Override
+    public NativeBytesStore<Void> zeroOut(long start, long end) {
+        return this;
+    }
+
+    @Override
+    public boolean compareAndSwapInt(long offset, int expected, int value) {
+        throw checkReadOnly();
+    }
+
+    @Override
+    public boolean compareAndSwapLong(long offset, long expected, long value) {
+        throw checkReadOnly();
+    }
+
+
+    @Override
+    public NativeBytesStore<Void> writeByte(long offset, byte i8) {
+        throw checkReadOnly();
+    }
+
+
+    @Override
+    public NativeBytesStore<Void> writeShort(long offset, short i16) {
+        throw checkReadOnly();
+    }
+
+
+    @Override
+    public NativeBytesStore<Void> writeInt(long offset, int i32) {
+        throw checkReadOnly();
+    }
+
+
+    @Override
+    public NativeBytesStore<Void> writeLong(long offset, long i64) {
+        throw checkReadOnly();
+    }
+
+
+    @Override
+    public NativeBytesStore<Void> writeOrderedLong(long offset, long i) {
+        throw checkReadOnly();
+    }
+
+
+    @Override
+    public NativeBytesStore<Void> writeFloat(long offset, float f) {
+        throw checkReadOnly();
+    }
+
+
+    @Override
+    public NativeBytesStore<Void> writeDouble(long offset, double d) {
+        throw checkReadOnly();
+    }
+
+
+    @Override
+    public NativeBytesStore<Void> writeVolatileByte(long offset, byte i8) {
+        throw checkReadOnly();
+    }
+
+
+    @Override
+    public NativeBytesStore<Void> writeVolatileShort(long offset, short i16) {
+        throw checkReadOnly();
+    }
+
+
+    @Override
+    public NativeBytesStore<Void> writeVolatileInt(long offset, int i32) {
+        throw checkReadOnly();
+    }
+
+
+    @Override
+    public NativeBytesStore<Void> writeVolatileLong(long offset, long i64) {
+        throw checkReadOnly();
+    }
+
+
+    @Override
+    public NativeBytesStore<Void> write(long offsetInRDO, byte[] bytes, int offset, int length) {
+        throw checkReadOnly();
+    }
+
+    @Override
+    public void write(long offsetInRDO,  ByteBuffer bytes, int offset, int length) {
+        throw checkReadOnly();
+    }
+
+
+    @Override
+    public NativeBytesStore<Void> write(long writeOffset,  RandomDataInput bytes, long readOffset, long length)
+            throws BufferOverflowException, BufferUnderflowException {
+        throw checkReadOnly();
+    }
+
+    @Override
+    public void write0(long offsetInRDO,  RandomDataInput bytes, long offset, long length) {
+        throw checkReadOnly();
+    }
+
+    @Override
+    public void nativeWrite(long address, long position, long size) {
+        throw checkReadOnly();
+    }
+
+    @Override
+    void write8bit(long position, char[] chars, int offset, int length) {
+        throw checkReadOnly();
+    }
+
+    @Override
+    public long appendUTF(long pos, char[] chars, int offset, int length) {
+        throw checkReadOnly();
+    }
+
+    @Override
+    public long appendUtf8(long pos, char[] chars, int offset, int length) {
+        throw checkReadOnly();
+    }
+
+
+    @Override
+    public VanillaBytes<Void> bytesForWrite() throws IllegalStateException {
+        throw checkReadOnly();
+    }
+
+
+    @Override
+    public NativeBytesStore<Void> writeOrderedInt(long offset, int i) {
+        throw checkReadOnly();
+    }
+
+    private IllegalStateException checkReadOnly() throws IllegalStateException {
+        throw new IllegalStateException("Read Only");
+    }
+
+    @Override
+    public boolean readWrite() {
+        return false;
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/ReleasedBytesStore.java

@@ -0,0 +1,319 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.io.ReferenceOwner;
+import org.jetbrains.annotations.NotNull;
+
+import java.nio.BufferOverflowException;
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+
+/**
+ * applied after a Bytes has been released and cannot be used.
+ *
+ * @see NoBytesStore
+ */
+@SuppressWarnings({"rawtypes", "unchecked"})
+public enum ReleasedBytesStore implements BytesStore {
+    RELEASED_BYTES_STORE;
+
+
+    public static <T, B extends BytesStore<B, T>> BytesStore<B, T> releasedBytesStore() {
+        return RELEASED_BYTES_STORE;
+    }
+
+    @Override
+    public void reserve(ReferenceOwner id) throws IllegalStateException {
+        throw newIllegalStateException();
+    }
+
+    @Override
+    public void release(ReferenceOwner id) throws IllegalStateException {
+        throw newIllegalStateException();
+    }
+
+    @Override
+    public int refCount() {
+        return 0;
+    }
+
+    @Override
+    public boolean tryReserve(ReferenceOwner id) throws IllegalStateException {
+        return false;
+    }
+
+    @Override
+    public boolean reservedBy(ReferenceOwner owner) {
+        return false;
+    }
+
+    @Override
+    public void releaseLast(ReferenceOwner id) throws IllegalStateException {
+        throw newIllegalStateException();
+    }
+
+
+    @Override
+    public RandomDataOutput writeByte(long offset, byte i8) {
+        throw newIllegalStateException();
+    }
+
+
+    private IllegalStateException newIllegalStateException() {
+        return new IllegalStateException("released");
+    }
+
+
+    @Override
+    public RandomDataOutput writeShort(long offset, short i) {
+        throw newIllegalStateException();
+    }
+
+
+    @Override
+    public RandomDataOutput writeInt(long offset, int i) {
+        throw newIllegalStateException();
+    }
+
+
+    @Override
+    public RandomDataOutput writeOrderedInt(long offset, int i) {
+        throw newIllegalStateException();
+    }
+
+
+    @Override
+    public RandomDataOutput writeLong(long offset, long i) {
+        throw newIllegalStateException();
+    }
+
+
+    @Override
+    public RandomDataOutput writeOrderedLong(long offset, long i) {
+        throw newIllegalStateException();
+    }
+
+
+    @Override
+    public RandomDataOutput writeFloat(long offset, float d) {
+        throw newIllegalStateException();
+    }
+
+
+    @Override
+    public RandomDataOutput writeDouble(long offset, double d) {
+        throw newIllegalStateException();
+    }
+
+
+    @Override
+    public RandomDataOutput writeVolatileByte(long offset, byte i8) {
+        throw newIllegalStateException();
+    }
+
+
+    @Override
+    public RandomDataOutput writeVolatileShort(long offset, short i16) {
+        throw newIllegalStateException();
+    }
+
+
+    @Override
+    public RandomDataOutput writeVolatileInt(long offset, int i32) {
+        throw newIllegalStateException();
+    }
+
+
+    @Override
+    public RandomDataOutput writeVolatileLong(long offset, long i64) {
+        throw newIllegalStateException();
+    }
+
+
+    @Override
+    public RandomDataOutput write(long offsetInRDO, byte[] bytes, int offset, int length) {
+        if (length != 0)
+            throw newIllegalStateException();
+        return this;
+    }
+
+    @Override
+    public void write(long offsetInRDO, ByteBuffer bytes, int offset, int length) {
+        throw newIllegalStateException();
+    }
+
+
+    @Override
+    public RandomDataOutput write(long writeOffset, RandomDataInput bytes, long readOffset, long length) {
+        throw newIllegalStateException();
+    }
+
+    @Override
+    public byte readByte(long offset) {
+        throw newIllegalStateException();
+    }
+
+    @Override
+    public int peekUnsignedByte(long offset) {
+        return -1;
+    }
+
+    @Override
+    public short readShort(long offset) {
+        throw newIllegalStateException();
+    }
+
+    @Override
+    public int readInt(long offset) {
+        throw newIllegalStateException();
+    }
+
+    @Override
+    public long readLong(long offset) {
+        throw newIllegalStateException();
+    }
+
+    @Override
+    public float readFloat(long offset) {
+        throw newIllegalStateException();
+    }
+
+    @Override
+    public double readDouble(long offset) {
+        throw newIllegalStateException();
+    }
+
+    @Override
+    public byte readVolatileByte(long offset) throws BufferUnderflowException {
+        throw newIllegalStateException();
+    }
+
+    @Override
+    public short readVolatileShort(long offset) throws BufferUnderflowException {
+        throw newIllegalStateException();
+    }
+
+    @Override
+    public int readVolatileInt(long offset) throws BufferUnderflowException {
+        throw newIllegalStateException();
+    }
+
+    @Override
+    public long readVolatileLong(long offset) throws BufferUnderflowException {
+        throw newIllegalStateException();
+    }
+
+    @Override
+    public boolean isDirectMemory() {
+        return false;
+    }
+
+
+    @Override
+    public BytesStore copy() {
+        return this;
+    }
+
+    @Override
+    public long capacity() {
+        return 0;
+    }
+
+    @Override
+    public Void underlyingObject() {
+        return null;
+    }
+
+    @Override
+    public boolean inside(long offset) {
+        return false;
+    }
+
+    @Override
+    public boolean inside(long offset, long buffer) {
+        return false;
+    }
+
+    @Override
+    public long copyTo( BytesStore store) {
+        throw newIllegalStateException();
+    }
+
+    @Override
+    public void nativeWrite(long address, long position, long size) {
+        throw newIllegalStateException();
+    }
+
+    @Override
+    public void nativeRead(long position, long address, long size) {
+        throw newIllegalStateException();
+    }
+
+    @Override
+    public boolean compareAndSwapInt(long offset, int expected, int value) {
+        throw newIllegalStateException();
+    }
+
+    @Override
+    public void testAndSetInt(long offset, int expected, int value) {
+        throw newIllegalStateException();
+    }
+
+    @Override
+    public boolean compareAndSwapLong(long offset, long expected, long value) {
+        throw newIllegalStateException();
+    }
+
+    @Override
+    public boolean equalBytes( BytesStore bytesStore, long length) {
+        throw newIllegalStateException();
+    }
+
+    @Override
+    public void move(long from, long to, long length) {
+        throw newIllegalStateException();
+    }
+
+    @Override
+    public long addressForRead(long offset) throws BufferUnderflowException {
+        throw newIllegalStateException();
+    }
+
+    @Override
+    public long addressForWrite(long offset) throws BufferOverflowException {
+        throw newIllegalStateException();
+    }
+
+    @Override
+    public long addressForWritePosition() throws UnsupportedOperationException, BufferOverflowException {
+        throw newIllegalStateException();
+    }
+
+
+    @Override
+    public Bytes bytesForWrite() {
+        throw newIllegalStateException();
+    }
+
+    @Override
+    public boolean sharedMemory() {
+        return false;
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/RingBufferReader.java

@@ -0,0 +1,69 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.io.Closeable;
+
+public interface RingBufferReader extends RingBufferReaderStats, Closeable {
+    long UNKNOWN_INDEX = -1;
+
+    boolean isEmpty();
+
+    boolean isStopped();
+
+    /**
+     * stop the reader. After being stopped, the reader will not block writers.
+     * After being stopped the reader can be re-opened
+     */
+    void stop();
+
+    /**
+     * the readPosition and readLimit will be adjusted so that the client can read the data
+     *
+     * @param bytes who's byteStore must be the ring buffer,
+     * @return nextReadPosition which should be passed to {@link RingBufferReader#afterRead(long)}
+     */
+    @SuppressWarnings("rawtypes")
+    long beforeRead(Bytes bytes);
+
+    void afterRead(long next);
+
+    void afterRead(long next, long payloadStart, long underlyingIndex);
+
+    long underlyingIndex();
+
+    /**
+     * Convenience method calls both {@link #beforeRead(Bytes)} and {@link #afterRead(long)}
+     *
+     * @param bytes
+     * @return whether read succeeded
+     */
+    @SuppressWarnings("rawtypes")
+    boolean read(BytesOut bytes);
+
+    /**
+     * @return the byteStore which backs the ring buffer
+     */
+    @SuppressWarnings("rawtypes")
+    BytesStore byteStore();
+
+    /**
+     * Take reader to just past the end of the RB
+     */
+    void toEnd();
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/RingBufferReaderStats.java

@@ -0,0 +1,27 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes;
+
+public interface RingBufferReaderStats {
+
+    long getAndClearReadCount();
+
+    long getAndClearMissedReadCount();
+
+    long behind();
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/StopCharTester.java

@@ -0,0 +1,42 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.bytes.util.EscapingStopCharTester;
+import org.jetbrains.annotations.NotNull;
+
+@FunctionalInterface
+public interface StopCharTester {
+    /**
+     * Detect which byte stops the string to be parsed
+     * <p>
+     * <p>This should be changed to support char instead.
+     * <p>
+     * <p>Note: for safety reasons, you should stop on a 0 byte or throw an IllegalStateException.
+     *
+     * @param ch to test, 0 should return true or throw an exception.
+     * @return if this byte is a stop character.
+     */
+    boolean isStopChar(int ch);
+
+
+    default StopCharTester escaping() {
+        return new EscapingStopCharTester(this);
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/StopCharTesters.java

@@ -0,0 +1,107 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+public enum StopCharTesters implements StopCharTester {
+    COMMA_STOP {
+        @Override
+        public boolean isStopChar(int ch) {
+            return ch < ' ' || ch == ',';
+        }
+    },
+    CURLY_STOP {
+        @Override
+        public boolean isStopChar(int ch) {
+            return ch < ' ' || ch == '}';
+        }
+    },
+    COMMA_SPACE_STOP {
+        @Override
+        public boolean isStopChar(int ch) {
+            return ch <= ' ' || ch == ',';
+        }
+    },
+    CONTROL_STOP {
+        @Override
+        public boolean isStopChar(int ch) {
+            return ch < ' ';
+        }
+    },
+    SPACE_STOP {
+        @Override
+        public boolean isStopChar(int ch) {
+            return Character.isWhitespace(ch) || ch == 0;
+        }
+    },
+    QUOTES {
+        @Override
+        public boolean isStopChar(int ch) {
+            return ch == '"' || ch <= 0;
+        }
+    },
+    SINGLE_QUOTES {
+        @Override
+        public boolean isStopChar(int ch) {
+            return ch == '\'' || ch <= 0;
+        }
+    },
+    EQUALS {
+        @Override
+        public boolean isStopChar(int ch) {
+            return ch == '=' || ch <= 0;
+        }
+    },
+    NUMBER_END {
+        @Override
+        public boolean isStopChar(int ch) {
+            switch (ch) {
+                case '0':
+                case '1':
+                case '2':
+                case '3':
+                case '4':
+                case '5':
+                case '6':
+                case '7':
+                case '8':
+                case '9':
+                case '+':
+                case '-':
+                case '.':
+                case 'E':
+                case 'e':
+                    return false;
+                default:
+                    return true;
+            }
+        }
+    },
+    NON_ALPHA_DIGIT {
+        @Override
+        public boolean isStopChar(int ch) {
+            return ch < '0' || !(Character.isAlphabetic(ch) || Character.isDigit(ch));
+        }
+    },
+    ALL {
+        @Override
+        public boolean isStopChar(int ch) {
+            return ch < 0;
+        }
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/StopCharsTester.java

@@ -0,0 +1,43 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.bytes.util.EscapingStopCharsTester;
+import org.jetbrains.annotations.NotNull;
+
+@FunctionalInterface
+public interface StopCharsTester {
+    /**
+     * Detect which byte or bytes stops the string to be parsed
+     * <p>
+     * <p>This should be changed to support char instead.
+     * <p>
+     * <p>Note: for safety reasons, you should stop on a 0 byte or throw an IllegalStateException.
+     *
+     * @param ch  to test, 0 should return true or throw an exception.
+     * @param ch2 to test, 0 should return true or throw an exception.
+     * @return if this byte is a stop character.
+     */
+    boolean isStopChar(int ch, int ch2);
+
+
+    default StopCharsTester escaping() {
+        return new EscapingStopCharsTester(this);
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/StreamingCommon.java

@@ -0,0 +1,32 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import org.jetbrains.annotations.NotNull;
+
+public interface StreamingCommon<S extends StreamingCommon<S>> extends RandomCommon {
+
+    /**
+     * Set the readPosition= writePosition = start, writeLimit = capacity
+     *
+     * @return this
+     */
+
+    S clear();
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/StreamingDataInput.java

@@ -0,0 +1,438 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.Jvm;
+import net.openhft.chronicle.core.Maths;
+import net.openhft.chronicle.core.OS;
+import net.openhft.chronicle.core.UnsafeMemory;
+import net.openhft.chronicle.core.io.IORuntimeException;
+import net.openhft.chronicle.core.util.Histogram;
+import net.openhft.chronicle.core.util.ThrowingConsumer;
+import net.openhft.chronicle.core.util.ThrowingConsumerNonCapturing;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.BufferOverflowException;
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+
+/**
+ * This data input has a a position() and a limit()
+ */
+@SuppressWarnings({"rawtypes", "unchecked"})
+public interface StreamingDataInput<S extends StreamingDataInput<S>> extends StreamingCommon<S> {
+
+    S readPosition(long position) throws BufferUnderflowException;
+
+
+    default S readPositionUnlimited(long position) throws BufferUnderflowException {
+        return readLimitToCapacity().readPosition(position);
+    }
+
+
+    default S readPositionRemaining(long position, long remaining) throws BufferUnderflowException {
+        readLimit(position + remaining);
+        return readPosition(position);
+    }
+
+
+    S readLimit(long limit) throws BufferUnderflowException;
+
+    default S readLimitToCapacity() throws BufferUnderflowException {
+        return readLimit(capacity());
+    }
+
+    /**
+     * Skip a number of bytes by moving the readPosition. Must be less than or equal to the readLimit.
+     *
+     * @param bytesToSkip bytes to skip.
+     * @return this
+     * @throws BufferUnderflowException if the offset is outside the limits of the Bytes
+     */
+
+    S readSkip(long bytesToSkip) throws BufferUnderflowException;
+
+    /**
+     * Read skip 1 when you are sure this is safe. Use at your own risk when you find a performance problem.
+     */
+    void uncheckedReadSkipOne();
+
+    /**
+     * Read skip -1 when you are sure this is safe. Use at your own risk when you find a performance problem.
+     */
+    void uncheckedReadSkipBackOne();
+
+    /**
+     * Perform a set of actions with a temporary bounds mode.
+     */
+    default void readWithLength0(long length,  ThrowingConsumerNonCapturing<S, IORuntimeException, BytesOut> bytesConsumer, StringBuilder sb, BytesOut toBytes)
+            throws BufferUnderflowException, IORuntimeException {
+        if (length > readRemaining())
+            throw new BufferUnderflowException();
+        long limit0 = readLimit();
+        long limit = readPosition() + length;
+        try {
+            readLimit(limit);
+            bytesConsumer.accept((S) this, sb, toBytes);
+        } finally {
+            readLimit(limit0);
+            readPosition(limit);
+        }
+    }
+
+    /**
+     * Perform a set of actions with a temporary bounds mode.
+     */
+    default void readWithLength(long length,  ThrowingConsumer<S, IORuntimeException> bytesConsumer)
+            throws BufferUnderflowException, IORuntimeException {
+        if (length > readRemaining())
+            throw new BufferUnderflowException();
+        long limit0 = readLimit();
+        long limit = readPosition() + length;
+        try {
+            readLimit(limit);
+            bytesConsumer.accept((S) this);
+        } finally {
+            readLimit(limit0);
+            readPosition(limit);
+        }
+    }
+
+
+    default InputStream inputStream() {
+        return new StreamingInputStream(this);
+    }
+
+    default long readStopBit() throws IORuntimeException {
+        return BytesInternal.readStopBit(this);
+    }
+
+    default char readStopBitChar() throws IORuntimeException {
+        return BytesInternal.readStopBitChar(this);
+    }
+
+    default double readStopBitDouble() {
+        return BytesInternal.readStopBitDouble(this);
+    }
+
+    default double readStopBitDecimal() throws BufferOverflowException {
+        long value = readStopBit();
+        int scale = (int) (Math.abs(value) % 10);
+        value /= 10;
+        return (double) value / Maths.tens(scale);
+    }
+
+    default boolean readBoolean() {
+        byte b = readByte();
+        return BytesUtil.byteToBoolean(b);
+    }
+
+    byte readByte();
+
+    default byte rawReadByte() {
+        return readByte();
+    }
+
+    default char readChar() {
+        return readStopBitChar();
+    }
+
+    /**
+     * @return the next unsigned 8 bit value or -1;
+     */
+    int readUnsignedByte();
+
+    /**
+     * @return the next unsigned 8 bit value or -1;
+     */
+    int uncheckedReadUnsignedByte();
+
+    short readShort() throws BufferUnderflowException;
+
+    default int readUnsignedShort() throws BufferUnderflowException {
+        return readShort() & 0xFFFF;
+    }
+
+    default int readInt24() throws BufferUnderflowException {
+        return readUnsignedShort() | (readUnsignedByte() << 24 >> 8);
+    }
+
+    default int readUnsignedInt24() throws BufferUnderflowException {
+        return readUnsignedShort() | (readUnsignedByte() << 16);
+    }
+
+    int readInt() throws BufferUnderflowException;
+
+    default int rawReadInt() {
+        return readInt();
+    }
+
+    default long readUnsignedInt()
+            throws BufferUnderflowException {
+        return readInt() & 0xFFFFFFFFL;
+    }
+
+    long readLong() throws BufferUnderflowException;
+
+    default long rawReadLong() {
+        return readLong();
+    }
+
+    /**
+     * @return a long using the bytes remaining
+     */
+    default long readIncompleteLong() {
+        long left = readRemaining();
+        try {
+            if (left >= 8)
+                return readLong();
+            if (left == 4)
+                return readInt();
+            long l = 0;
+            for (int i = 0, remaining = (int) left; i < remaining; i++) {
+                l |= (long) readUnsignedByte() << (i * 8);
+            }
+            return l;
+
+        } catch (BufferUnderflowException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+    float readFloat() throws BufferUnderflowException;
+
+    double readDouble() throws BufferUnderflowException;
+
+    /**
+     * The same as readUTF() except the length is stop bit encoded.  This saves one byte for strings shorter than 128
+     * chars.  <code>null</code> values are also supported
+     *
+     * @return a Unicode string or <code>null</code> if <code>writeUtf8(null)</code> was called
+     */
+
+    default String readUtf8()
+            throws BufferUnderflowException, IORuntimeException, IllegalArgumentException {
+        return BytesInternal.readUtf8(this);
+    }
+
+
+    @Deprecated(/* to be removed in x.22 */)
+    default String readUTFΔ()
+            throws IORuntimeException, BufferUnderflowException, IllegalArgumentException {
+        return BytesInternal.readUtf8(this);
+    }
+
+
+    default String read8bit() throws IORuntimeException, BufferUnderflowException {
+        return BytesInternal.read8bit(this);
+    }
+
+    /**
+     * The same as readUtf8() except the chars are copied to a truncated StringBuilder.
+     *
+     * @param sb to copy chars to
+     * @return <code>true</code> if there was a String, or <code>false</code> if it was <code>null</code>
+     */
+    default <ACS extends Appendable & CharSequence> boolean readUtf8( ACS sb)
+            throws IORuntimeException, IllegalArgumentException, BufferUnderflowException {
+        AppendableUtil.setLength(sb, 0);
+        if (readRemaining() <= 0)
+            // TODO throw BufferUnderflowException here? please review
+            return false;
+        long len0 = readStopBit();
+        if (len0 == -1)
+            return false;
+        int len = Maths.toUInt31(len0);
+        if (len > 0)
+            BytesInternal.parseUtf8(this, sb, true, len);
+        return true;
+    }
+
+    @Deprecated(/* to be removed in x.22 */)
+    default <ACS extends Appendable & CharSequence> boolean readUTFΔ( ACS sb)
+            throws IORuntimeException, IllegalArgumentException, BufferUnderflowException {
+        return readUtf8(sb);
+    }
+
+    default boolean read8bit( Bytes b)
+            throws BufferUnderflowException, IllegalStateException, BufferOverflowException {
+        b.clear();
+        if (readRemaining() <= 0)
+            return false;
+        long len0 = this.readStopBit();
+        if (len0 == -1)
+            return false;
+        int len = Maths.toUInt31(len0);
+        b.write((BytesStore) this, readPosition(), len);
+        readSkip(len);
+        return true;
+    }
+
+    default <ACS extends Appendable & CharSequence> boolean read8bit( ACS sb)
+            throws IORuntimeException, IllegalArgumentException, BufferUnderflowException {
+        AppendableUtil.setLength(sb, 0);
+        long len0 = BytesInternal.readStopBit(this);
+        if (len0 == -1)
+            return false;
+        int len = Maths.toUInt31(len0);
+        try {
+            AppendableUtil.parse8bit(this, sb, len);
+        } catch (IOException e) {
+            throw new IORuntimeException(e);
+        }
+        return true;
+    }
+
+    default boolean read8bit( StringBuilder sb)
+            throws IORuntimeException, BufferUnderflowException {
+        sb.setLength(0);
+        long len0 = BytesInternal.readStopBit(this);
+        if (len0 == -1)
+            return false;
+        int len = Maths.toUInt31(len0);
+        try {
+            AppendableUtil.parse8bit(this, sb, len);
+        } catch (IOException e) {
+            throw new IORuntimeException(e);
+        }
+        return true;
+    }
+
+    default int read( byte[] bytes) {
+        return read(bytes, 0, bytes.length);
+    }
+
+    default int read( byte[] bytes, int off, int len) {
+        long remaining = readRemaining();
+        if (remaining <= 0)
+            return -1;
+        int len2 = (int) Math.min(len, remaining);
+        int i = 0;
+        for (; i < len2 - 7; i += 8)
+            UnsafeMemory.unsafePutLong(bytes, i + off, rawReadLong());
+        for (; i < len2; i++)
+            bytes[off + i] = rawReadByte();
+        return len2;
+    }
+
+    default int read( char[] bytes, int off, int len) {
+        long remaining = readRemaining();
+        if (remaining <= 0)
+            return -1;
+        int len2 = (int) Math.min(len, remaining);
+        for (int i = 0; i < len2; i++)
+            bytes[off + i] = (char) readUnsignedByte();
+        return len2;
+    }
+
+    default void read( ByteBuffer buffer) {
+        for (int i = (int) Math.min(readRemaining(), buffer.remaining()); i > 0; i--)
+            buffer.put(readByte());
+    }
+
+    default void read( Bytes bytes, int length) {
+        int len2 = (int) Math.min(length, readRemaining());
+        int i = 0;
+        for (; i < len2 - 7; i += 8)
+            bytes.rawWriteLong(rawReadLong());
+        for (; i < len2; i++)
+            bytes.rawWriteByte(rawReadByte());
+    }
+
+    default void unsafeReadObject(Object o, int length) {
+        unsafeReadObject(o, (o.getClass().isArray() ? 4 : 0) + Jvm.objectHeaderSize(), length);
+    }
+
+    default void unsafeReadObject(Object o, int offset, int length) {
+        assert BytesUtil.isTriviallyCopyable(o.getClass(), offset, length);
+        if (readRemaining() < length)
+            throw new BufferUnderflowException();
+        int i = 0;
+        for (; i < length - 7; i += 8)
+            UnsafeMemory.unsafePutLong(o, offset + i, rawReadLong());
+        for (; i < length; i++)
+            UnsafeMemory.unsafePutByte(o, offset + i, rawReadByte());
+    }
+
+    int readVolatileInt() throws BufferUnderflowException;
+
+    long readVolatileLong() throws BufferUnderflowException;
+
+    int peekUnsignedByte();
+
+
+
+    default <E extends Enum<E>> E readEnum( Class<E> eClass)
+            throws IORuntimeException, BufferUnderflowException {
+        return BytesInternal.readEnum(this, eClass);
+    }
+
+    @Deprecated(/* to be removed in x.22 */)
+    default void parseUTF(Appendable sb, int length)
+            throws IllegalArgumentException, BufferUnderflowException, UTFDataFormatRuntimeException {
+        parseUtf8(sb, length);
+    }
+
+    default void parseUtf8(Appendable sb, int encodedLength)
+            throws IllegalArgumentException, BufferUnderflowException, UTFDataFormatRuntimeException {
+        parseUtf8(sb, true, encodedLength);
+    }
+
+    default void parseUtf8(Appendable sb, boolean utf, int length)
+            throws IllegalArgumentException, BufferUnderflowException, UTFDataFormatRuntimeException {
+        AppendableUtil.setLength(sb, 0);
+        BytesInternal.parseUtf8(this, sb, utf, length);
+    }
+
+    default long parseHexLong() {
+        return BytesInternal.parseHexLong(this);
+    }
+
+    void copyTo(OutputStream out) throws IOException;
+
+    long copyTo(BytesStore to);
+
+    default void readHistogram( Histogram histogram) {
+        BytesInternal.readHistogram(this, histogram);
+    }
+
+    default void readWithLength(Bytes bytes) {
+        bytes.clear();
+        int length = Maths.toUInt31(readStopBit());
+        int i;
+        for (i = 0; i < length - 7; i++)
+            bytes.writeLong(readLong());
+        for (; i < length; i++)
+            bytes.writeByte(readByte());
+    }
+
+    /**
+     * When there is no more data to read, return zero, false and empty string.
+     *
+     * @param lenient if true, return nothing rather than error.
+     */
+    void lenient(boolean lenient);
+
+    boolean lenient();
+
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/StreamingDataOutput.java

@@ -0,0 +1,562 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.Jvm;
+import net.openhft.chronicle.core.Maths;
+import net.openhft.chronicle.core.UnsafeMemory;
+import net.openhft.chronicle.core.annotation.Java9;
+import net.openhft.chronicle.core.util.Histogram;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.BufferOverflowException;
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+
+/**
+ * Position based access.  Once data has been read, the position() moves.
+ * <p>The use of this instance is single threaded, though the use of the data
+ */
+@SuppressWarnings({"rawtypes", "unchecked"})
+public interface StreamingDataOutput<S extends StreamingDataOutput<S>> extends StreamingCommon<S> {
+    int JAVA9_STRING_CODER_LATIN = 0;
+    int JAVA9_STRING_CODER_UTF16 = 1;
+
+    S writePosition(long position) throws BufferOverflowException;
+
+    S writeLimit(long limit) throws BufferOverflowException;
+
+    /**
+     * Skip a number of bytes by moving the writePosition. Must be less than or equal to the writeLimit.
+     *
+     * @param bytesToSkip bytes to skip.
+     * @return this
+     * @throws BufferOverflowException if the offset is outside the limits of the Bytes
+     */
+    S writeSkip(long bytesToSkip) throws BufferOverflowException;
+
+    default S alignBy(int width) {
+        return writeSkip((-writePosition()) & (width - 1));
+    }
+
+    /**
+     * @return Bytes as an OutputStream
+     */
+    
+    default OutputStream outputStream() {
+        return new StreamingOutputStream(this);
+    }
+
+    /**
+     * Write a stop bit encoded long
+     *
+     * @param x long to write
+     * @return this.
+     */
+    
+    default S writeStopBit(long x) throws BufferOverflowException {
+        BytesInternal.writeStopBit(this, x);
+        return (S) this;
+    }
+
+    
+    default S writeStopBit(char x) throws BufferOverflowException {
+        BytesInternal.writeStopBit(this, x);
+        return (S) this;
+    }
+
+    
+    default S writeStopBit(double d) throws BufferOverflowException {
+        BytesInternal.writeStopBit(this, d);
+        return (S) this;
+    }
+
+    
+    default S writeStopBitDecimal(double d) throws BufferOverflowException {
+        boolean negative = d < 0;
+        double ad = Math.abs(d);
+        long value;
+        int scale = 0;
+        if ((long) ad == ad) {
+            value = (long) ad * 10;
+
+        } else {
+            double factor = 1;
+            while (scale < 9) {
+                double v = ad * factor;
+                if (v >= 1e14 || (long) v == v)
+                    break;
+                factor *= 10;
+                scale++;
+            }
+            value = Math.round(ad * factor);
+            while (scale > 0 && value % 10 == 0) {
+                value /= 10;
+                scale--;
+            }
+            value = value * 10 + scale;
+        }
+        if (negative)
+            value = -value;
+        BytesInternal.writeStopBit(this, value);
+        return (S) this;
+    }
+
+    /**
+     * Write the same encoding as <code>writeUTF</code> with the following changes.  1) The length is stop bit encoded
+     * i.e. one byte longer for short strings, but is not limited in length. 2) The string can be null.
+     *
+     * @param cs the string value to be written. Can be null.
+     * @throws BufferOverflowException if there is not enough space left
+     */
+    
+    default S writeUtf8(CharSequence cs)
+            throws BufferOverflowException {
+        BytesInternal.writeUtf8(this, cs);
+        return (S) this;
+    }
+
+    
+    default S writeUtf8(String s)
+            throws BufferOverflowException {
+        BytesInternal.writeUtf8(this, s);
+        return (S) this;
+    }
+
+    
+    @Deprecated(/* to be removed in x.22 */)
+    default S writeUTFΔ(CharSequence cs) throws BufferOverflowException {
+        return writeUtf8(cs);
+    }
+
+    
+    default S write8bit(CharSequence cs)
+            throws BufferOverflowException {
+        if (cs == null)
+            return writeStopBit(-1);
+
+        if (cs instanceof BytesStore)
+            return write8bit((BytesStore) cs);
+
+        if (cs instanceof String)
+            return write8bit((String) cs);
+
+        return write8bit(cs, 0, cs.length());
+    }
+
+    
+    default S write8bit( CharSequence s, int start, int length)
+            throws BufferOverflowException, IllegalArgumentException, IndexOutOfBoundsException {
+        writeStopBit(length);
+        for (int i = 0; i < length; i++) {
+            char c = s.charAt(i + start);
+            rawWriteByte((byte) Maths.toUInt8((int) c));
+        }
+        return (S) this;
+    }
+
+    
+    default S write(CharSequence cs)
+            throws BufferOverflowException, BufferUnderflowException, IllegalArgumentException {
+        if (cs instanceof BytesStore) {
+            return write((BytesStore) cs);
+        }
+        return write(cs, 0, cs.length());
+    }
+
+    
+    default S write( CharSequence s, int start, int length)
+            throws BufferOverflowException, IllegalArgumentException, IndexOutOfBoundsException {
+        for (int i = 0; i < length; i++) {
+            char c = s.charAt(i + start);
+            appendUtf8(c);
+        }
+        return (S) this;
+    }
+
+    
+    default S write8bit(String s)
+            throws BufferOverflowException {
+        if (s == null)
+            writeStopBit(-1);
+        else
+            write8bit(s, 0, (int) Math.min(writeRemaining(), s.length()));
+        return (S) this;
+    }
+
+    
+    default S write8bit(BytesStore bs)
+            throws BufferOverflowException {
+        if (bs == null) {
+            writeStopBit(-1);
+        } else {
+            long offset = bs.readPosition();
+            long readRemaining = Math.min(writeRemaining(), bs.readLimit() - offset);
+            writeStopBit(readRemaining);
+            write(bs, offset, readRemaining);
+        }
+        return (S) this;
+    }
+
+    
+    S writeByte(byte i8) throws BufferOverflowException;
+
+    default S rawWriteByte(byte i8) throws BufferOverflowException {
+        return writeByte(i8);
+    }
+
+    
+    default S writeUnsignedByte(int i)
+            throws BufferOverflowException, IllegalArgumentException {
+        return writeByte((byte) Maths.toUInt8(i));
+    }
+
+    
+    default S writeChar(char ch) {
+        return writeStopBit(ch);
+    }
+
+    
+    S writeShort(short i16) throws BufferOverflowException;
+
+    
+    default S writeUnsignedShort(int u16)
+            throws BufferOverflowException, IllegalArgumentException {
+        return writeShort((short) Maths.toUInt16(u16));
+    }
+
+    
+    default S writeInt24(int i) throws BufferOverflowException {
+        writeUnsignedShort((short) i);
+        return writeUnsignedByte((i >>> 16) & 0xFF);
+    }
+
+    
+    default S writeUnsignedInt24(int i) throws BufferOverflowException {
+        writeUnsignedShort((short) i);
+        return writeUnsignedByte(i >>> 16);
+    }
+
+    
+    S writeInt(int i) throws BufferOverflowException;
+
+    default S rawWriteInt(int i) throws BufferOverflowException {
+        return writeInt(i);
+    }
+
+    
+    S writeIntAdv(int i, int advance) throws BufferOverflowException;
+
+    
+    default S writeUnsignedInt(long i)
+            throws BufferOverflowException, IllegalArgumentException {
+        return writeInt((int) Maths.toUInt32(i));
+    }
+
+    /**
+     * Write a long
+     */
+    
+    S writeLong(long i64) throws BufferOverflowException;
+
+    /**
+     * Write a long without a bounds check
+     */
+    default S rawWriteLong(long i) throws BufferOverflowException {
+        return writeLong(i);
+    }
+
+    
+    S writeLongAdv(long i64, int advance) throws BufferOverflowException;
+
+    
+    S writeFloat(float f) throws BufferOverflowException;
+
+    
+    S writeDouble(double d) throws BufferOverflowException;
+
+    
+    S writeDoubleAndInt(double d, int i) throws BufferOverflowException;
+
+    /**
+     * Write all data or fail.
+     */
+    
+    default S write( RandomDataInput bytes) {
+        assert bytes != this : "you should not write to yourself !";
+
+        try {
+            return write(bytes, bytes.readPosition(), Math.min(writeRemaining(), bytes.readRemaining()));
+        } catch (BufferOverflowException | BufferUnderflowException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+    /**
+     * Writes the passed BytesStore
+     *
+     * @param bytes to write
+     * @return this
+     */
+    default S write( BytesStore bytes) {
+        assert bytes != this : "you should not write to yourself !";
+
+        try {
+            return write(bytes, bytes.readPosition(), Math.min(writeRemaining(), bytes.readRemaining()));
+        } catch (BufferOverflowException | BufferUnderflowException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+    /**
+     * @return capacity without resize
+     */
+    long realCapacity();
+
+    /**
+     * @return writeRemaining with resize
+     */
+    long realWriteRemaining();
+
+    default boolean canWriteDirect(long count) {
+        return false;
+    }
+
+    
+    default S writeSome( Bytes bytes) {
+        try {
+            long length = Math.min(bytes.readRemaining(), writeRemaining());
+            if (length + writePosition() >= 1 << 20)
+                length = Math.min(bytes.readRemaining(), realCapacity() - writePosition());
+            write(bytes, bytes.readPosition(), length);
+            if (length == bytes.readRemaining()) {
+                bytes.clear();
+            } else {
+                bytes.readSkip(length);
+                if (bytes.writePosition() > bytes.realCapacity() / 2)
+                    bytes.compact();
+            }
+            return (S) this;
+        } catch (BufferOverflowException | BufferUnderflowException | IllegalArgumentException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+    /**
+     * Write all data or fail.
+     */
+    
+    default S write( RandomDataInput bytes, long offset, long length)
+            throws BufferOverflowException, BufferUnderflowException {
+        BytesInternal.writeFully(bytes, offset, length, this);
+        return (S) this;
+    }
+
+    /**
+     * Write all data or fail.
+     */
+    
+    default S write( BytesStore bytes, long offset, long
+            length)
+            throws BufferOverflowException, BufferUnderflowException {
+        if (length + writePosition() > capacity())
+            throw new IllegalArgumentException("Cannot write " + length + " bytes as position is " + writePosition() + " and capacity is " + capacity());
+        BytesInternal.writeFully(bytes, offset, length, this);
+        return (S) this;
+    }
+
+    
+    default S write( byte[] bytes) throws BufferOverflowException {
+        write(bytes, 0, bytes.length);
+        return (S) this;
+    }
+
+    /**
+     * Write all data or fail.
+     */
+    
+    S write(byte[] bytes, int offset, int length) throws BufferOverflowException;
+
+    default S unsafeWriteObject(Object o, int length) {
+        return unsafeWriteObject(o, (o.getClass().isArray() ? 4 : 0) + Jvm.objectHeaderSize(), length);
+    }
+
+    default S unsafeWriteObject(Object o, int offset, int length) {
+        int i = 0;
+        for (; i < length - 7; i += 8)
+            writeLong(UnsafeMemory.unsafeGetLong(o, offset + i));
+        for (; i < length; i++)
+            writeByte(UnsafeMemory.unsafeGetByte(o, offset + i));
+        return (S) this;
+    }
+
+    
+    S writeSome(ByteBuffer buffer) throws BufferOverflowException;
+
+    
+    default S writeBoolean(boolean flag) throws BufferOverflowException {
+        return writeByte(flag ? (byte) 'Y' : (byte) 'N');
+    }
+
+    
+    S writeOrderedInt(int i) throws BufferOverflowException;
+
+    
+    S writeOrderedLong(long i) throws BufferOverflowException;
+
+    default <E extends Enum<E>> S writeEnum( E e)
+            throws BufferOverflowException {
+        return write8bit(e.name());
+    }
+
+    
+    default S appendUtf8( CharSequence cs)
+            throws BufferOverflowException {
+        return appendUtf8(cs, 0, cs.length());
+    }
+
+    
+    default S appendUtf8(int codepoint) throws BufferOverflowException {
+        BytesInternal.appendUtf8Char(this, codepoint);
+        return (S) this;
+    }
+
+    
+    default S appendUtf8(char[] chars, int offset, int length)
+            throws BufferOverflowException, IllegalArgumentException {
+        int i;
+        ascii:
+        {
+            for (i = 0; i < length; i++) {
+                char c = chars[offset + i];
+                if (c > 0x007F)
+                    break ascii;
+                writeByte((byte) c);
+            }
+            return (S) this;
+        }
+        for (; i < length; i++) {
+            char c = chars[offset + i];
+            BytesInternal.appendUtf8Char(this, c);
+        }
+        return (S) this;
+    }
+
+    
+    default S appendUtf8( CharSequence cs, int offset, int length)
+            throws BufferOverflowException, IllegalArgumentException {
+        BytesInternal.appendUtf8(this, cs, offset, length);
+        return (S) this;
+    }
+
+    // length is number of characters (not bytes)
+    @Java9
+    
+    default S appendUtf8(byte[] bytes, int offset, int length, byte coder)
+            throws BufferOverflowException, IllegalArgumentException {
+        if (coder == JAVA9_STRING_CODER_LATIN) {
+            for (int i = 0; i < length; i++) {
+                byte b = bytes[offset + i];
+                int b2 = (b & 0xFF);
+                BytesInternal.appendUtf8Char(this, b2);
+            }
+        } else {
+            assert coder == JAVA9_STRING_CODER_UTF16;
+            for (int i = 0; i < 2 * length; i += 2) {
+                byte b1 = bytes[2 * offset + i];
+                byte b2 = bytes[2 * offset + i + 1];
+
+                int uBE = ((b2 & 0xFF) << 8) | b1 & 0xFF;
+                BytesInternal.appendUtf8Char(this, uBE);
+            }
+        }
+        return (S) this;
+    }
+
+    @Java9
+    
+    default S appendUtf8(byte[] bytes, int offset, int length)
+            throws BufferOverflowException, IllegalArgumentException {
+        for (int i = 0; i < length; i++) {
+            int b = bytes[offset + i] & 0xFF; // unsigned byte
+
+            if (b >= 0xF0) {
+                int b2 = bytes[offset + i + 1] & 0xFF; // unsigned byte
+                int b3 = bytes[offset + i + 2] & 0xFF; // unsigned byte
+                int b4 = bytes[offset + i + 3] & 0xFF; // unsigned byte
+                this.writeByte((byte) b4);
+                this.writeByte((byte) b3);
+                this.writeByte((byte) b2);
+                this.writeByte((byte) b);
+
+                i += 3;
+            } else if (b >= 0xE0) {
+                int b2 = bytes[offset + i + 1] & 0xFF; // unsigned byte
+                int b3 = bytes[offset + i + 2] & 0xFF; // unsigned byte
+                this.writeByte((byte) b3);
+                this.writeByte((byte) b2);
+                this.writeByte((byte) b);
+
+                i += 2;
+            } else if (b >= 0xC0) {
+                int b2 = bytes[offset + i + 1] & 0xFF; // unsigned byte
+                this.writeByte((byte) b2);
+                this.writeByte((byte) b);
+
+                i += 1;
+            } else {
+                this.writeByte((byte) b);
+            }
+        }
+        return (S) this;
+    }
+
+    default void copyFrom( InputStream input) throws IOException, BufferOverflowException, IllegalArgumentException {
+        BytesInternal.copy(input, this);
+    }
+
+    default void writePositionRemaining(long position, long length) {
+        writeLimit(position + length);
+        writePosition(position);
+    }
+
+    default void writeHistogram( Histogram histogram) {
+        BytesInternal.writeHistogram(this, histogram);
+    }
+
+    default void writeBigDecimal( BigDecimal bd) {
+        writeBigInteger(bd.unscaledValue());
+        writeStopBit(bd.scale());
+    }
+
+    default void writeBigInteger( BigInteger bi) {
+        byte[] bytes = bi.toByteArray();
+        writeStopBit(bytes.length);
+        write(bytes);
+    }
+
+    default void writeWithLength(RandomDataInput bytes) {
+        writeStopBit(bytes.readRemaining());
+        write(bytes);
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/StreamingInputStream.java

@@ -0,0 +1,70 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import org.jetbrains.annotations.NotNull;
+
+import java.io.IOException;
+import java.io.InputStream;
+
+@SuppressWarnings("rawtypes")
+public class StreamingInputStream extends InputStream {
+
+    private StreamingDataInput in;
+
+    public StreamingInputStream() {
+        this(NoBytesStore.NO_BYTES);
+    }
+
+    public StreamingInputStream(StreamingDataInput in) {
+        this.in = in;
+    }
+
+
+    public StreamingInputStream init(StreamingDataInput in) {
+        this.in = in;
+        return this;
+    }
+
+    @Override
+    public long skip(long n) throws IOException {
+        long len = Math.min(in.readRemaining(), n);
+        in.readSkip(len);
+        return len;
+    }
+
+    @Override
+    public int available() throws IOException {
+        return (int) Math.min(Integer.MAX_VALUE, in.readRemaining());
+    }
+
+    @Override
+    public int read(byte[] b, int off, int len) throws IOException {
+        if (len == 0) {
+            return 0;
+        }
+        int len2 = in.read(b, off, len);
+        return len2 == 0 ? -1 : len2;
+    }
+
+    @Override
+    public int read() throws IOException {
+        return in.readRemaining() > 0 ? in.readUnsignedByte() : -1;
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/StreamingOutputStream.java

@@ -0,0 +1,64 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import org.jetbrains.annotations.NotNull;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.BufferOverflowException;
+
+@SuppressWarnings("rawtypes")
+public class StreamingOutputStream extends OutputStream {
+    private StreamingDataOutput sdo;
+
+    public StreamingOutputStream() {
+        this(NoBytesStore.NO_BYTES);
+    }
+
+    public StreamingOutputStream(StreamingDataOutput sdo) {
+        this.sdo = sdo;
+    }
+
+
+    public StreamingOutputStream init(StreamingDataOutput sdo) {
+        this.sdo = sdo;
+        return this;
+    }
+
+    @Override
+    public void write(byte[] b, int off, int len) throws IOException {
+        try {
+            sdo.write(b, off, len);
+
+        } catch ( BufferOverflowException | IllegalArgumentException e) {
+            throw new IOException(e);
+        }
+    }
+
+    @Override
+    public void write(int b) throws IOException {
+        try {
+            sdo.writeUnsignedByte(0xff & b);
+
+        } catch ( BufferOverflowException | IllegalArgumentException e) {
+            throw new IOException(e);
+        }
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/SubBytes.java

@@ -0,0 +1,58 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import org.jetbrains.annotations.NotNull;
+
+@SuppressWarnings({"rawtypes", "unchecked"})
+public class SubBytes<Underlying> extends VanillaBytes<Underlying> {
+    private final long start;
+    private final long capacity;
+
+    public SubBytes( BytesStore bytesStore, long start, long capacity) throws IllegalStateException {
+        super(bytesStore);
+        this.start = start;
+        this.capacity = capacity;
+        clear();
+        readLimit(writeLimit());
+    }
+
+    public SubBytes( BytesStore bytesStore) throws IllegalStateException {
+        super(bytesStore);
+        this.start = 0;
+        this.capacity = bytesStore.capacity();
+        clear();
+        readLimit(writeLimit());
+    }
+
+    @Override
+    public long capacity() {
+        return capacity;
+    }
+
+    @Override
+    public long start() {
+        return start;
+    }
+
+    @Override
+    public long realCapacity() {
+        return capacity;
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/UTFDataFormatRuntimeException.java

@@ -0,0 +1,31 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.io.IORuntimeException;
+
+public class UTFDataFormatRuntimeException extends IORuntimeException {
+    public UTFDataFormatRuntimeException(String message) {
+        super(message);
+    }
+
+    public UTFDataFormatRuntimeException(String message, Exception cause) {
+        super(message, cause);
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/UncheckedBytes.java

@@ -0,0 +1,294 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.Jvm;
+import net.openhft.chronicle.core.OS;
+import net.openhft.chronicle.core.util.StringUtils;
+import org.jetbrains.annotations.NotNull;
+
+import java.nio.BufferOverflowException;
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+
+import static net.openhft.chronicle.core.util.StringUtils.extractBytes;
+import static net.openhft.chronicle.core.util.StringUtils.extractChars;
+
+/**
+ * Fast unchecked version of AbstractBytes
+ */
+@SuppressWarnings({"rawtypes", "unchecked"})
+public class UncheckedBytes<Underlying>
+        extends AbstractBytes<Underlying> {
+    Bytes underlyingBytes;
+
+    public UncheckedBytes( Bytes underlyingBytes) throws IllegalStateException {
+        super(underlyingBytes.bytesStore(), underlyingBytes.writePosition(), underlyingBytes.writeLimit());
+        this.underlyingBytes = underlyingBytes;
+        readPosition(underlyingBytes.readPosition());
+    }
+
+    public void setBytes( Bytes bytes) throws IllegalStateException {
+        BytesStore underlyingBytes = bytes.bytesStore();
+        if (bytesStore != underlyingBytes) {
+            bytesStore.release(this);
+            this.bytesStore(underlyingBytes);
+            bytesStore.reserve(this);
+        }
+        readPosition(bytes.readPosition());
+        this.uncheckedWritePosition(bytes.writePosition());
+        this.writeLimit = bytes.writeLimit();
+
+        this.underlyingBytes = bytes;
+    }
+
+    @Override
+    public void ensureCapacity(long size) throws IllegalArgumentException {
+        if (size > realCapacity()) {
+            underlyingBytes.ensureCapacity(size);
+            bytesStore(underlyingBytes.bytesStore());
+        }
+    }
+
+    @Override
+
+    public Bytes<Underlying> unchecked(boolean unchecked) {
+        return this;
+    }
+
+    @Override
+    public boolean unchecked() {
+        return true;
+    }
+
+    @Override
+    void writeCheckOffset(long offset, long adding) {
+    }
+
+    @Override
+    void readCheckOffset(long offset, long adding, boolean given) {
+    }
+
+    @Override
+    void prewriteCheckOffset(long offset, long subtracting) {
+    }
+
+
+    @Override
+    public Bytes<Underlying> readPosition(long position) {
+        readPosition = position;
+        return this;
+    }
+
+
+    @Override
+    public Bytes<Underlying> readLimit(long limit) {
+        uncheckedWritePosition(limit);
+        return this;
+    }
+
+
+    @Override
+    public Bytes<Underlying> writePosition(long position) {
+        uncheckedWritePosition(position);
+        return this;
+    }
+
+
+    @Override
+    public Bytes<Underlying> readSkip(long bytesToSkip) {
+        readPosition += bytesToSkip;
+        return this;
+    }
+
+
+    @Override
+    public Bytes<Underlying> writeSkip(long bytesToSkip) {
+        uncheckedWritePosition(writePosition() + bytesToSkip);
+        return this;
+    }
+
+
+    @Override
+    public Bytes<Underlying> writeLimit(long limit) {
+        writeLimit = limit;
+        return this;
+    }
+
+
+    @Override
+    public BytesStore<Bytes<Underlying>, Underlying> copy() {
+        throw new UnsupportedOperationException("todo");
+    }
+
+    @Override
+    public boolean isElastic() {
+        return false;
+    }
+
+    @Override
+    protected long readOffsetPositionMoved(long adding) {
+        long offset = readPosition;
+        readPosition += adding;
+        return offset;
+    }
+
+    @Override
+    protected long writeOffsetPositionMoved(long adding, long advance) {
+        long oldPosition = writePosition();
+        uncheckedWritePosition(writePosition() + advance);
+        return oldPosition;
+    }
+
+    @Override
+    protected long prewriteOffsetPositionMoved(long subtracting) throws BufferOverflowException {
+        return readPosition -= subtracting;
+    }
+
+
+    @Override
+    public Bytes<Underlying> write( RandomDataInput bytes, long offset, long length)
+            throws BufferOverflowException, IllegalArgumentException {
+        if (length == 8) {
+            writeLong(bytes.readLong(offset));
+
+        } else {
+            super.write(bytes, offset, length);
+        }
+        return this;
+    }
+
+
+    public Bytes<Underlying> write( BytesStore bytes, long offset, long length)
+            throws BufferOverflowException, IllegalArgumentException {
+        if (length == 8) {
+            writeLong(bytes.readLong(offset));
+        } else if (bytes.underlyingObject() == null
+                && bytesStore
+                .isDirectMemory() &&
+                length >= 32) {
+            rawCopy(bytes, offset, length);
+
+        } else {
+            super.write(bytes, offset, length);
+        }
+        return this;
+    }
+
+    @Override
+
+    public Bytes<Underlying> append8bit( CharSequence cs)
+            throws BufferOverflowException, BufferUnderflowException {
+        if (cs instanceof RandomDataInput) {
+            return write((RandomDataInput) cs);
+        }
+
+        int length = cs.length();
+        long offset = writeOffsetPositionMoved(length);
+        for (int i = 0; i < length; i++) {
+            char c = cs.charAt(i);
+            if (c > 255) c = '?';
+            writeByte(offset, (byte) c);
+        }
+        return this;
+    }
+
+    long rawCopy( BytesStore bytes, long offset, long length)
+            throws BufferOverflowException, IllegalArgumentException {
+        long len = Math.min(writeRemaining(), Math.min(bytes.capacity() - offset, length));
+        if (len > 0) {
+            writeCheckOffset(writePosition(), len);
+            this.throwExceptionIfReleased();
+            OS.memory().copyMemory(bytes.addressForRead(offset), addressForWritePosition(), len);
+            writeSkip(len);
+        }
+        return len;
+    }
+
+
+    @Override
+    public Bytes<Underlying> writeByte(byte i8) throws BufferOverflowException {
+        long offset = writeOffsetPositionMoved(1, 1);
+        bytesStore.writeByte(offset, i8);
+        return this;
+    }
+
+
+    @Override
+    public Bytes<Underlying> writeUtf8(String s) throws BufferOverflowException {
+        if (s == null) {
+            writeStopBit(-1);
+            return this;
+        }
+
+        try {
+            if (Jvm.isJava9Plus()) {
+                byte[] strBytes = extractBytes(s);
+                byte coder = StringUtils.getStringCoder(s);
+                long utfLength = AppendableUtil.findUtf8Length(strBytes, coder);
+                writeStopBit(utfLength);
+                appendUtf8(strBytes, 0, s.length(), coder);
+            } else {
+                char[] chars = extractChars(s);
+                long utfLength = AppendableUtil.findUtf8Length(chars);
+                writeStopBit(utfLength);
+                if (utfLength == chars.length)
+                    append8bit(chars);
+                else
+                    appendUtf8(chars, 0, chars.length);
+            }
+        } catch (IllegalArgumentException e) {
+            throw new AssertionError(e);
+        }
+        return this;
+    }
+
+    void append8bit(char[] chars) throws BufferOverflowException, IllegalArgumentException {
+        for (int i = 0; i < chars.length; i++) {
+            char c = chars[i];
+            bytesStore.writeByte(writePosition++, (byte) c);
+        }
+    }
+
+
+    @Override
+    public Bytes<Underlying> appendUtf8(char[] chars, int offset, int length) throws BufferOverflowException, IllegalArgumentException {
+        int i;
+        ascii:
+        {
+            for (i = 0; i < length; i++) {
+                char c = chars[offset + i];
+                if (c > 0x007F)
+                    break ascii;
+                bytesStore.writeByte(writePosition++, (byte) c);
+            }
+            return this;
+        }
+        for (; i < length; i++) {
+            char c = chars[offset + i];
+            BytesInternal.appendUtf8Char(this, c);
+        }
+        return this;
+    }
+
+    @Override
+    public void write(long offsetInRDO, ByteBuffer bytes, int offset, int length) throws BufferOverflowException {
+
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/UncheckedNativeBytes.java

@@ -0,0 +1,930 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.bytes.algo.BytesStoreHash;
+import net.openhft.chronicle.core.Jvm;
+import net.openhft.chronicle.core.Memory;
+import net.openhft.chronicle.core.OS;
+import net.openhft.chronicle.core.annotation.ForceInline;
+import net.openhft.chronicle.core.io.AbstractReferenceCounted;
+import net.openhft.chronicle.core.io.BackgroundResourceReleaser;
+import net.openhft.chronicle.core.io.IORuntimeException;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+import java.nio.BufferOverflowException;
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+
+/**
+ * Fast unchecked version of AbstractBytes
+ */
+@SuppressWarnings({"rawtypes", "unchecked"})
+public class UncheckedNativeBytes<Underlying>
+        extends AbstractReferenceCounted
+        implements Bytes<Underlying> {
+    protected final long capacity;
+
+    private final Bytes<Underlying> underlyingBytes;
+
+    protected NativeBytesStore<Underlying> bytesStore;
+    protected long readPosition;
+    protected long writePosition;
+    protected long writeLimit;
+    private int lastDecimalPlaces = 0;
+
+    public UncheckedNativeBytes( Bytes<Underlying> underlyingBytes)
+            throws IllegalStateException {
+        this.underlyingBytes = underlyingBytes;
+        underlyingBytes.reserve(this);
+        this.bytesStore = (NativeBytesStore<Underlying>) underlyingBytes.bytesStore();
+        assert bytesStore.start() == 0;
+        writePosition = underlyingBytes.writePosition();
+        writeLimit = underlyingBytes.writeLimit();
+        readPosition = underlyingBytes.readPosition();
+        capacity = bytesStore.capacity();
+    }
+
+    @Override
+    public void ensureCapacity(long size) throws IllegalArgumentException {
+        if (size > realCapacity()) {
+            underlyingBytes.ensureCapacity(size);
+            bytesStore = (NativeBytesStore<Underlying>) underlyingBytes.bytesStore();
+        }
+    }
+
+    @Override
+    public boolean unchecked() {
+        return true;
+    }
+
+    @Override
+    public boolean isDirectMemory() {
+        return true;
+    }
+
+    @Override
+
+    public Bytes<Underlying> unchecked(boolean unchecked) {
+        return this;
+    }
+
+    @Override
+    public void move(long from, long to, long length) {
+        bytesStore.move(from - start(), to - start(), length);
+    }
+
+
+    @Override
+    public Bytes<Underlying> compact() {
+        long start = start();
+        long readRemaining = readRemaining();
+        if (readRemaining > 0 && start < readPosition) {
+            bytesStore.move(readPosition, start, readRemaining);
+            readPosition = start;
+            writePosition = readPosition + readRemaining;
+        }
+        return this;
+    }
+
+
+    @Override
+    public Bytes<Underlying> readPosition(long position) {
+        readPosition = position;
+        return this;
+    }
+
+
+    @Override
+    public Bytes<Underlying> readLimit(long limit) {
+        writePosition = limit;
+        return this;
+    }
+
+
+    @Override
+    public Bytes<Underlying> writePosition(long position) {
+        writePosition = position;
+        return this;
+    }
+
+
+    @Override
+    public Bytes<Underlying> readSkip(long bytesToSkip) {
+        readPosition += bytesToSkip;
+        return this;
+    }
+
+    @Override
+    public byte readVolatileByte(long offset) throws BufferUnderflowException {
+        return bytesStore.readVolatileByte(offset);
+    }
+
+    @Override
+    public short readVolatileShort(long offset) throws BufferUnderflowException {
+        return bytesStore.readVolatileShort(offset);
+    }
+
+    @Override
+    public int readVolatileInt(long offset) throws BufferUnderflowException {
+        return bytesStore.readVolatileInt(offset);
+    }
+
+    @Override
+    public long readVolatileLong(long offset) throws BufferUnderflowException {
+        return bytesStore.readVolatileLong(offset);
+    }
+
+    @Override
+    public void uncheckedReadSkipOne() {
+        readPosition++;
+    }
+
+    @Override
+    public void uncheckedReadSkipBackOne() {
+        readPosition--;
+    }
+
+
+    @Override
+    public Bytes<Underlying> writeSkip(long bytesToSkip) {
+        writePosition += bytesToSkip;
+        return this;
+    }
+
+
+    @Override
+    public Bytes<Underlying> writeLimit(long limit) {
+        writeLimit = limit;
+        return this;
+    }
+
+
+    @Override
+    public BytesStore<Bytes<Underlying>, Underlying> copy() {
+        throw new UnsupportedOperationException("todo");
+    }
+
+    @Override
+    public boolean isElastic() {
+        return false;
+    }
+
+    protected long readOffsetPositionMoved(long adding) {
+        long offset = readPosition;
+        readPosition += adding;
+        return offset;
+    }
+
+    protected long writeOffsetPositionMoved(long adding) {
+        return writeOffsetPositionMoved(adding, adding);
+    }
+
+    protected long writeOffsetPositionMoved(long adding, long advance) {
+        long oldPosition = writePosition;
+        long writeEnd = oldPosition + adding;
+        assert writeEnd <= bytesStore.safeLimit();
+        writePosition += advance;
+        return oldPosition;
+    }
+
+    protected long prewriteOffsetPositionMoved(long substracting) {
+        return readPosition -= substracting;
+    }
+
+
+    @Override
+    public Bytes<Underlying> write( RandomDataInput bytes, long offset, long length)
+            throws BufferUnderflowException, BufferOverflowException {
+        if (length == 8) {
+            writeLong(bytes.readLong(offset));
+
+        } else if (length >= 16 && bytes.isDirectMemory()) {
+            rawCopy(bytes, offset, length);
+
+        } else {
+            BytesInternal.writeFully(bytes, offset, length, this);
+        }
+        return this;
+    }
+
+    public long rawCopy( RandomDataInput bytes, long offset, long length)
+            throws BufferOverflowException, BufferUnderflowException {
+        long len = Math.min(writeRemaining(), Math.min(bytes.capacity() - offset, length));
+        if (len > 0) {
+            writeCheckOffset(writePosition(), len);
+            this.throwExceptionIfReleased();
+            OS.memory().copyMemory(bytes.addressForRead(offset), addressForWritePosition(), len);
+            writeSkip(len);
+        }
+        return len;
+    }
+
+    @Override
+
+    public Bytes<Underlying> clear() {
+        readPosition = writePosition = start();
+        writeLimit = capacity();
+        return this;
+    }
+
+
+    @Override
+    public Bytes<Underlying> clearAndPad(long length) throws BufferOverflowException {
+        if (start() + length > capacity())
+            throw new BufferOverflowException();
+        readPosition = writePosition = start() + length;
+        writeLimit = capacity();
+        return this;
+    }
+
+    @Override
+    @ForceInline
+    public long readLimit() {
+        return writePosition;
+    }
+
+    @Override
+    @ForceInline
+    public long writeLimit() {
+        return writeLimit;
+    }
+
+    @Override
+    @ForceInline
+    public long realCapacity() {
+        return bytesStore.realCapacity();
+    }
+
+    @Override
+    public long realWriteRemaining() {
+        return writeRemaining();
+    }
+
+    @Override
+    @ForceInline
+    public long capacity() {
+        return capacity;
+    }
+
+
+    @Override
+    public Underlying underlyingObject() {
+        return bytesStore.underlyingObject();
+    }
+
+    @Override
+    @ForceInline
+    public long readPosition() {
+        return readPosition;
+    }
+
+    @Override
+    @ForceInline
+    public long writePosition() {
+        return writePosition;
+    }
+
+    @Override
+    @ForceInline
+    public boolean compareAndSwapInt(long offset, int expected, int value)
+            throws BufferOverflowException {
+        writeCheckOffset(offset, 4);
+        return bytesStore.compareAndSwapInt(offset, expected, value);
+    }
+
+    @Override
+    public void testAndSetInt(long offset, int expected, int value) {
+        writeCheckOffset(offset, 4);
+        bytesStore.testAndSetInt(offset, expected, value);
+    }
+
+    @Override
+    @ForceInline
+    public boolean compareAndSwapLong(long offset, long expected, long value)
+            throws BufferOverflowException {
+        writeCheckOffset(offset, 8);
+        return bytesStore.compareAndSwapLong(offset, expected, value);
+    }
+
+    @Override
+    protected void performRelease() {
+        this.underlyingBytes.release(this);
+        // need to wait as checks rely on this completing.
+        BackgroundResourceReleaser.releasePendingResources();
+    }
+
+    @Override
+    public int readUnsignedByte() {
+        long offset = readOffsetPositionMoved(1);
+        return bytesStore.memory.readByte(bytesStore.address + offset) & 0xFF;
+    }
+
+    @Override
+    public int uncheckedReadUnsignedByte() {
+        return readUnsignedByte();
+    }
+
+    @Override
+    @ForceInline
+    public byte readByte() {
+        long offset = readOffsetPositionMoved(1);
+        return bytesStore.memory.readByte(bytesStore.address + offset);
+    }
+
+    @Override
+    @ForceInline
+    public int peekUnsignedByte() {
+        try {
+            return readRemaining() > 0 ? bytesStore.readUnsignedByte(readPosition) : -1;
+
+        } catch (BufferUnderflowException e) {
+            return -1;
+        }
+    }
+
+    @Override
+    @ForceInline
+    public short readShort() {
+        long offset = readOffsetPositionMoved(2);
+        return bytesStore.readShort(offset);
+    }
+
+    @Override
+    @ForceInline
+    public int readInt() {
+        long offset = readOffsetPositionMoved(4);
+        return bytesStore.readInt(offset);
+    }
+
+    @Override
+    @ForceInline
+    public long readLong() {
+        long offset = readOffsetPositionMoved(8);
+        return bytesStore.readLong(offset);
+    }
+
+    @Override
+    @ForceInline
+    public float readFloat() {
+        long offset = readOffsetPositionMoved(4);
+        return bytesStore.readFloat(offset);
+    }
+
+    @Override
+    @ForceInline
+    public double readDouble() {
+        long offset = readOffsetPositionMoved(8);
+        return bytesStore.readDouble(offset);
+    }
+
+    @Override
+    @ForceInline
+    public int readVolatileInt() {
+        long offset = readOffsetPositionMoved(4);
+        return bytesStore.readVolatileInt(offset);
+    }
+
+    @Override
+    @ForceInline
+    public long readVolatileLong() {
+        long offset = readOffsetPositionMoved(8);
+        return bytesStore.readVolatileLong(offset);
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> writeByte(long offset, byte i) throws BufferOverflowException {
+        writeCheckOffset(offset, 1);
+        bytesStore.writeByte(offset, i);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> writeShort(long offset, short i) throws BufferOverflowException {
+        writeCheckOffset(offset, 2);
+        bytesStore.writeShort(offset, i);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> writeInt(long offset, int i) throws BufferOverflowException {
+        writeCheckOffset(offset, 4);
+        bytesStore.writeInt(offset, i);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> writeOrderedInt(long offset, int i) throws BufferOverflowException {
+        writeCheckOffset(offset, 4);
+        bytesStore.writeOrderedInt(offset, i);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> writeLong(long offset, long i) throws BufferOverflowException {
+        writeCheckOffset(offset, 8);
+        bytesStore.writeLong(offset, i);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> writeOrderedLong(long offset, long i) throws BufferOverflowException {
+        writeCheckOffset(offset, 8);
+        bytesStore.writeOrderedLong(offset, i);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> writeFloat(long offset, float d) throws BufferOverflowException {
+        writeCheckOffset(offset, 4);
+        bytesStore.writeFloat(offset, d);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> writeDouble(long offset, double d) throws BufferOverflowException {
+        writeCheckOffset(offset, 8);
+        bytesStore.writeDouble(offset, d);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> writeVolatileByte(long offset, byte i8)
+            throws BufferOverflowException {
+        writeCheckOffset(offset, 1);
+        bytesStore.writeVolatileByte(offset, i8);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> writeVolatileShort(long offset, short i16)
+            throws BufferOverflowException {
+        writeCheckOffset(offset, 2);
+        bytesStore.writeVolatileShort(offset, i16);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> writeVolatileInt(long offset, int i32)
+            throws BufferOverflowException {
+        writeCheckOffset(offset, 4);
+        bytesStore.writeVolatileInt(offset, i32);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> writeVolatileLong(long offset, long i64)
+            throws BufferOverflowException {
+        writeCheckOffset(offset, 8);
+        bytesStore.writeVolatileLong(offset, i64);
+        return this;
+    }
+
+    @Override
+
+    @ForceInline
+    public Bytes<Underlying> write(long offsetInRDO, byte[] bytes, int offset, int length)
+            throws BufferOverflowException {
+        writeCheckOffset(offsetInRDO, length);
+        bytesStore.write(offsetInRDO, bytes, offset, length);
+        return this;
+    }
+
+    @Override
+    @ForceInline
+    public void write(long offsetInRDO,  ByteBuffer bytes, int offset, int length) throws BufferOverflowException {
+        writeCheckOffset(offsetInRDO, length);
+        bytesStore.write(offsetInRDO, bytes, offset, length);
+    }
+
+    @Override
+
+    @ForceInline
+    public Bytes<Underlying> write(long writeOffset,
+                                    RandomDataInput bytes, long readOffset, long length)
+            throws BufferUnderflowException, BufferOverflowException {
+        writeCheckOffset(writeOffset, length);
+        bytesStore.write(writeOffset, bytes, readOffset, length);
+        return this;
+    }
+
+    @ForceInline
+    void writeCheckOffset(long offset, long adding) throws BufferOverflowException {
+//        assert writeCheckOffset0(offset, adding);
+    }
+
+    @Override
+    @ForceInline
+    public byte readByte(long offset) {
+        return bytesStore.readByte(offset);
+    }
+
+    @Override
+    public int readUnsignedByte(long offset) {
+        return bytesStore.memory.readByte(bytesStore.address + offset) & 0xFF;
+    }
+
+    @Override
+    public int peekUnsignedByte(long offset) {
+        return offset >= writePosition ? -1 : readByte(offset);
+    }
+
+    @Override
+    @ForceInline
+    public short readShort(long offset) {
+        return bytesStore.readShort(offset);
+    }
+
+    @Override
+    @ForceInline
+    public int readInt(long offset) {
+        return bytesStore.readInt(offset);
+    }
+
+    @Override
+    @ForceInline
+    public long readLong(long offset) {
+        return bytesStore.readLong(offset);
+    }
+
+    @Override
+    @ForceInline
+    public float readFloat(long offset) {
+        return bytesStore.readFloat(offset);
+    }
+
+    @Override
+    @ForceInline
+    public double readDouble(long offset) {
+        return bytesStore.readDouble(offset);
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> writeByte(byte i8) {
+        long offset = writeOffsetPositionMoved(1);
+        bytesStore.memory.writeByte(bytesStore.address + offset, i8);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> prewriteByte(byte i8) {
+        long offset = prewriteOffsetPositionMoved(1);
+        bytesStore.memory.writeByte(bytesStore.address + offset, i8);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> writeShort(short i16) {
+        long offset = writeOffsetPositionMoved(2);
+        bytesStore.writeShort(offset, i16);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> prewriteShort(short i16) {
+        long offset = prewriteOffsetPositionMoved(2);
+        bytesStore.writeShort(offset, i16);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> writeInt(int i) {
+        long offset = writeOffsetPositionMoved(4);
+        bytesStore.writeInt(offset, i);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> writeIntAdv(int i, int advance) {
+        long offset = writeOffsetPositionMoved(4, advance);
+        bytesStore.writeInt(offset, i);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> prewriteInt(int i) {
+        long offset = prewriteOffsetPositionMoved(4);
+        bytesStore.writeInt(offset, i);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> writeLong(long i64) {
+        long offset = writeOffsetPositionMoved(8);
+        bytesStore.writeLong(offset, i64);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> writeLongAdv(long i64, int advance) {
+        long offset = writeOffsetPositionMoved(8, advance);
+        bytesStore.writeLong(offset, i64);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> prewriteLong(long i64) {
+        long offset = prewriteOffsetPositionMoved(8);
+        bytesStore.writeLong(offset, i64);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> writeFloat(float f) {
+        long offset = writeOffsetPositionMoved(4);
+        bytesStore.writeFloat(offset, f);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> writeDouble(double d) {
+        long offset = writeOffsetPositionMoved(8);
+        bytesStore.writeDouble(offset, d);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> writeDoubleAndInt(double d, int i) {
+        long offset = writeOffsetPositionMoved(12);
+        bytesStore.writeDouble(offset, d);
+        bytesStore.writeInt(offset + 8, i);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> write( byte[] bytes, int offset, int length) {
+        if (length + offset > bytes.length)
+            throw new ArrayIndexOutOfBoundsException("bytes.length=" + bytes.length + ", " +
+                    "length=" + length + ", offset=" + offset);
+        if (length > writeRemaining())
+            throw new BufferOverflowException();
+        long offsetInRDO = writeOffsetPositionMoved(length);
+        bytesStore.write(offsetInRDO, bytes, offset, length);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> prewrite( byte[] bytes) {
+        long offsetInRDO = prewriteOffsetPositionMoved(bytes.length);
+        bytesStore.write(offsetInRDO, bytes);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> prewrite( BytesStore bytes) {
+        long offsetInRDO = prewriteOffsetPositionMoved(bytes.length());
+        bytesStore.write(offsetInRDO, bytes);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> writeSome( ByteBuffer buffer) {
+        bytesStore.write(writePosition, buffer, buffer.position(), buffer.limit());
+        writePosition += buffer.remaining();
+        assert writePosition <= writeLimit();
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> writeOrderedInt(int i) {
+        long offset = writeOffsetPositionMoved(4);
+        bytesStore.writeOrderedInt(offset, i);
+        return this;
+    }
+
+
+    @Override
+    @ForceInline
+    public Bytes<Underlying> writeOrderedLong(long i) {
+        long offset = writeOffsetPositionMoved(8);
+        bytesStore.writeOrderedLong(offset, i);
+        return this;
+    }
+
+    @Override
+    public long addressForRead(long offset) throws BufferUnderflowException {
+        return bytesStore.addressForRead(offset);
+    }
+
+    @Override
+    public long addressForWrite(long offset) throws BufferOverflowException {
+        return bytesStore.addressForWrite(offset);
+    }
+
+    @Override
+    public long addressForWritePosition() throws UnsupportedOperationException, BufferOverflowException {
+        return bytesStore.addressForWrite(0);
+    }
+
+    @Override
+    public int hashCode() {
+        return BytesStoreHash.hash32(this);
+    }
+
+    @Override
+    public boolean equals(Object obj) {
+        if (!(obj instanceof Bytes)) return false;
+         Bytes b2 = (Bytes) obj;
+        long remaining = readRemaining();
+        return b2.readRemaining() == remaining && equalsBytes(b2, remaining);
+    }
+
+    public boolean equalsBytes( Bytes b2, long remaining) {
+        long i = 0;
+        try {
+            for (; i < remaining - 7; i += 8)
+                if (readLong(readPosition() + i) != b2.readLong(b2.readPosition() + i))
+                    return false;
+            for (; i < remaining; i++)
+                if (readByte(readPosition() + i) != b2.readByte(b2.readPosition() + i))
+                    return false;
+
+        } catch (BufferUnderflowException e) {
+            throw Jvm.rethrow(e);
+        }
+        return true;
+    }
+
+
+    @Override
+    public String toString() {
+        return BytesInternal.toString(this);
+    }
+
+
+    @Override
+    public void lenient(boolean lenient) {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public boolean lenient() {
+        return false;
+    }
+
+
+
+    @Override
+    @ForceInline
+    public void nativeRead(long position, long address, long size) {
+        bytesStore.nativeRead(position, address, size);
+    }
+
+    @Override
+    @ForceInline
+    public void nativeWrite(long address, long position, long size) {
+        bytesStore.nativeWrite(address, position, size);
+    }
+
+
+    @Override
+    public BytesStore bytesStore() {
+        return bytesStore;
+    }
+
+    @Override
+    public int byteCheckSum() throws IORuntimeException {
+         NativeBytesStore bytesStore = (NativeBytesStore) bytesStore();
+        return bytesStore.byteCheckSum(readPosition(), readLimit());
+    }
+
+    @Override
+
+    public Bytes<Underlying> append8bit( CharSequence cs)
+            throws BufferOverflowException, BufferUnderflowException {
+        if (cs instanceof BytesStore) {
+            return write((BytesStore) cs);
+        }
+        int length = cs.length();
+        long offset = writeOffsetPositionMoved(length);
+        long address = bytesStore.addressForWrite(offset);
+         Memory memory = bytesStore.memory;
+        assert memory != null;
+        try {
+            int i = 0;
+            for (; i < length - 1; i += 2) {
+                char c = cs.charAt(i);
+                char c2 = cs.charAt(i + 1);
+                memory.writeByte(address + i, (byte) c);
+                memory.writeByte(address + i + 1, (byte) c2);
+            }
+            for (; i < length; i++) {
+                char c = cs.charAt(i);
+                memory.writeByte(address + i, (byte) c);
+            }
+
+            return this;
+        } catch (IndexOutOfBoundsException e) {
+            throw new AssertionError(e);
+        }
+    }
+
+
+    @Override
+    public Bytes<Underlying> appendUtf8(char[] chars, int offset, int length) throws BufferOverflowException, IllegalArgumentException {
+        ensureCapacity(length);
+         NativeBytesStore nbs = this.bytesStore;
+        long position = nbs.appendUtf8(writePosition(), chars, offset, length);
+        writePosition(position);
+        return this;
+    }
+
+    @Override
+    public int lastDecimalPlaces() {
+        return lastDecimalPlaces;
+    }
+
+    @Override
+    public void lastDecimalPlaces(int lastDecimalPlaces) {
+        this.lastDecimalPlaces = Math.max(0, lastDecimalPlaces);
+    }
+
+
+    @Override
+    public Bytes<Underlying> write( RandomDataInput bytes) {
+        assert bytes != this : "you should not write to yourself !";
+
+        try {
+            return write(bytes, bytes.readPosition(), Math.min(writeRemaining(), bytes.readRemaining()));
+        } catch (BufferOverflowException | BufferUnderflowException e) {
+            throw new AssertionError(e);
+        }
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/UpdateInterceptor.java

@@ -0,0 +1,18 @@
+package net.openhft.chronicle.bytes;
+
+/**
+ * Represents an operation that accepts a single input argument then modifies that same instance.
+ */
+@FunctionalInterface
+public interface UpdateInterceptor {
+
+    /**
+     * modifies {@code t} with changed data
+     *
+     * @param methodName the name of the method
+     * @param t          the input argument - for a method call with multiple arguments, the last one is passed
+     * @return whether to proceed. If false, don't write
+     */
+    boolean update(String methodName, Object t);
+
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/VanillaBytes.java

@@ -0,0 +1,597 @@
+    /*
+     * Copyright 2016-2020 chronicle.software
+     *
+     * https://chronicle.software
+     *
+     * Licensed under the Apache License, Version 2.0 (the "License");
+     * you may not use this file except in compliance with the License.
+     * You may obtain a copy of the License at
+     *
+     *     http://www.apache.org/licenses/LICENSE-2.0
+     *
+     * Unless required by applicable law or agreed to in writing, software
+     * distributed under the License is distributed on an "AS IS" BASIS,
+     * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+     * See the License for the specific language governing permissions and
+     * limitations under the License.
+     */
+
+    package net.openhft.chronicle.bytes;
+
+    import net.openhft.chronicle.core.Jvm;
+    import net.openhft.chronicle.core.Maths;
+    import net.openhft.chronicle.core.Memory;
+    import net.openhft.chronicle.core.OS;
+    import net.openhft.chronicle.core.annotation.Java9;
+    import net.openhft.chronicle.core.io.IORuntimeException;
+    import net.openhft.chronicle.core.util.StringUtils;
+    import org.jetbrains.annotations.NotNull;
+    import org.jetbrains.annotations.Nullable;
+
+    import java.nio.BufferOverflowException;
+    import java.nio.BufferUnderflowException;
+    import java.nio.ByteBuffer;
+
+    import static net.openhft.chronicle.bytes.NoBytesStore.noBytesStore;
+
+    /**
+     * Simple Bytes implementation which is not Elastic.
+     */
+    @SuppressWarnings({"rawtypes", "unchecked"})
+    public class VanillaBytes<Underlying>
+            extends AbstractBytes<Underlying>
+            implements Byteable<Bytes<Underlying>, Underlying>, Comparable<CharSequence> {
+
+        public VanillaBytes( BytesStore bytesStore) throws IllegalStateException {
+            this(bytesStore, bytesStore.writePosition(), bytesStore.writeLimit());
+        }
+
+        public VanillaBytes( BytesStore bytesStore, long writePosition, long writeLimit) throws IllegalStateException {
+            super(bytesStore, writePosition, writeLimit);
+        }
+
+        /**
+         * @return a non elastic bytes.
+         */
+
+        public static VanillaBytes<Void> vanillaBytes() {
+            try {
+                return new VanillaBytes<>(noBytesStore());
+
+            } catch (IllegalStateException e) {
+                throw new AssertionError(e);
+            }
+        }
+
+        @Java9
+        private static boolean isEqual0( byte[] bytes, byte coder,  NativeBytesStore bs, long address) {
+
+             Memory memory = bs.memory;
+
+            if (coder == 0) {
+                int i = 0;
+                for (; i < bytes.length; i++) {
+                    byte b = memory.readByte(address + i);
+                    char c = (char) (bytes[i] & 0xFF);
+                    if (b != c) {
+                        return false;
+                    }
+                }
+            } else {
+                int i = 0;
+                for (; i < bytes.length; i++) {
+                    byte b = memory.readByte(address + i);
+                    char c = (char) (((bytes[i + 1] & 0xFF) << 8) | (bytes[i] & 0xFF));
+
+                    if (b != c) {
+                        return false;
+                    }
+                }
+            }
+
+            return true;
+        }
+
+        private static boolean isEqual0( char[] chars,  NativeBytesStore bs, long address) {
+
+             Memory memory = bs.memory;
+            int i = 0;
+            for (; i < chars.length - 3; i += 4) {
+                int b = memory.readInt(address + i);
+                int b0 = b & 0xFF;
+                int b1 = (b >> 8) & 0xFF;
+                int b2 = (b >> 16) & 0xFF;
+                int b3 = (b >> 24) & 0xFF;
+                if (b0 != chars[i] || b1 != chars[i + 1] || b2 != chars[i + 2] || b3 != chars[i + 3])
+                    return false;
+            }
+            for (; i < chars.length; i++) {
+                int b = memory.readByte(address + i) & 0xFF;
+                if (b != chars[i])
+                    return false;
+            }
+
+            return true;
+        }
+
+        private static boolean isEqual1( char[] chars,  BytesStore bytesStore, long readPosition) throws BufferUnderflowException {
+            for (int i = 0; i < chars.length; i++) {
+                int b = bytesStore.readByte(readPosition + i) & 0xFF;
+                if (b != chars[i])
+                    return false;
+            }
+            return true;
+        }
+
+        @Java9
+        private static boolean isEqual1( byte[] bytes, byte coder,  BytesStore bytesStore, long readPosition) throws BufferUnderflowException {
+            for (int i = 0; i < bytes.length; i++) {
+                int b = bytesStore.readByte(readPosition + i) & 0xFF;
+                char c;
+
+                if (coder == 0) {
+                    c = (char) (bytes[i] & 0xFF);
+                } else {
+                    c = (char) (((bytes[i + 1] & 0xFF) << 8) | (bytes[i] & 0xFF));
+                    i++;
+                }
+
+                if (b != c)
+                    return false;
+            }
+            return true;
+        }
+
+        @Override
+        public long readVolatileLong(long offset) throws BufferUnderflowException {
+            readCheckOffset(offset, 8, true);
+            return bytesStore.readVolatileLong(offset);
+        }
+
+        @Override
+        public void bytesStore( BytesStore<Bytes<Underlying>, Underlying> byteStore, long offset, long length)
+                throws IllegalStateException, BufferOverflowException, BufferUnderflowException {
+            setBytesStore(byteStore);
+            // assume its read-only
+            readLimit(offset + length);
+            writeLimit(offset + length);
+            readPosition(offset);
+        }
+
+        private void setBytesStore( BytesStore<Bytes<Underlying>, Underlying> bytesStore)
+                throws IllegalStateException {
+            if (this.bytesStore != bytesStore) {
+                 BytesStore oldBS = this.bytesStore;
+                this.bytesStore(bytesStore);
+                bytesStore.reserve(this);
+                oldBS.release(this);
+            }
+            clear();
+        }
+
+        @Override
+        public long offset() {
+            return readPosition();
+        }
+
+        @Override
+        public long maxSize() {
+            return readRemaining();
+        }
+
+        @Override
+        public boolean isElastic() {
+            return false;
+        }
+
+
+        @Override
+        public Bytes<Underlying> bytesForRead() throws IllegalStateException {
+            return isClear()
+                    ? new VanillaBytes<>(bytesStore, writePosition(), bytesStore.writeLimit())
+                    : new SubBytes<>(bytesStore, readPosition(), readLimit());
+        }
+
+        @Override
+        public boolean isEqual( String s) {
+            if (s == null || s.length() != readRemaining()) return false;
+
+            try {
+                if (Jvm.isJava9Plus()) {
+                    byte[] bytes = StringUtils.extractBytes(s);
+                    byte coder = StringUtils.getStringCoder(s);
+                    if (bytesStore instanceof NativeBytesStore) {
+                         NativeBytesStore bs = (NativeBytesStore) this.bytesStore;
+                        long address = bs.addressForRead(readPosition);
+                        return isEqual0(bytes, coder, bs, address);
+
+                    } else {
+                        return isEqual1(bytes, coder, bytesStore, readPosition);
+                    }
+                } else {
+                    char[] chars = StringUtils.extractChars(s);
+                    if (bytesStore instanceof NativeBytesStore) {
+                         NativeBytesStore bs = (NativeBytesStore) this.bytesStore;
+                        long address = bs.addressForRead(readPosition);
+                        return isEqual0(chars, bs, address);
+
+                    } else {
+                        return isEqual1(chars, bytesStore, readPosition);
+                    }
+                }
+            } catch (BufferUnderflowException e) {
+                throw new AssertionError(e);
+            }
+        }
+
+        @Override
+        public long realCapacity() {
+            return bytesStore.realCapacity();
+        }
+
+
+        @Override
+        public BytesStore<Bytes<Underlying>, Underlying> copy() {
+            if (bytesStore.underlyingObject() instanceof ByteBuffer) {
+                try {
+                    ByteBuffer bb = ByteBuffer.allocateDirect(Maths.toInt32(readRemaining()));
+                     ByteBuffer bbu = (ByteBuffer) bytesStore.underlyingObject();
+                    ByteBuffer slice = bbu.slice();
+                    slice.position((int) readPosition());
+                    slice.limit((int) readLimit());
+                    bb.put(slice);
+                    bb.clear();
+                    return (BytesStore) BytesStore.wrap(bb);
+                } catch (IllegalArgumentException e) {
+                    throw new AssertionError(e);
+                }
+            } else {
+                return (BytesStore) NativeBytes.copyOf(this);
+            }
+        }
+
+
+        @Override
+        public Bytes<Underlying> write( RandomDataInput bytes, long offset, long length)
+                throws BufferOverflowException, BufferUnderflowException, IllegalArgumentException {
+            optimisedWrite(bytes, offset, length);
+            return this;
+        }
+
+        protected void optimisedWrite( RandomDataInput bytes, long offset, long length) {
+            if (length <= safeCopySize() && isDirectMemory() && bytes.isDirectMemory()) {
+                long len = Math.min(writeRemaining(), Math.min(bytes.capacity() - offset, length));
+                if (len > 0) {
+                    writeCheckOffset(writePosition(), len);
+                    long address = bytes.addressForRead(offset);
+                    long address2 = addressForWritePosition();
+                    assert address != 0;
+                    assert address2 != 0;
+                    OS.memory().copyMemory(address, address2, len);
+                    writeSkip(len);
+                }
+            } else {
+                super.write(bytes, offset, length);
+            }
+        }
+
+        public void write(long position,  CharSequence str, int offset, int length)
+                throws BufferOverflowException, IllegalArgumentException {
+
+            ensureCapacity(length);
+            if (offset + length > str.length())
+                throw new IllegalArgumentException("offset=" + offset + " + length=" + length + " > str.length =" + str.length());
+
+            for (int i = 0; i < length; i++) {
+                bytesStore.writeByte(position + i, str.charAt(offset + i));
+            }
+        }
+
+        @Override
+
+        public VanillaBytes append( CharSequence str, int start, int end) throws IndexOutOfBoundsException {
+            assert end > start : "end=" + end + ",start=" + start;
+            try {
+                if (isDirectMemory()) {
+                    if (str instanceof BytesStore) {
+
+                        write((BytesStore) str, (long) start, end - start);
+                        return this;
+                    }
+                    if (str instanceof String) {
+                        if (Jvm.isJava9Plus()) {
+                            byte coder = StringUtils.getStringCoder((String) str);
+                            appendUtf8(StringUtils.extractBytes((String) str), start, end - start, coder);
+                        } else {
+                            appendUtf8(StringUtils.extractChars((String) str), start, end - start);
+                        }
+
+                        return this;
+                    }
+                }
+                super.append(str, start, end);
+                return this;
+
+            } catch (Exception e) {
+                throw new IndexOutOfBoundsException(e.toString());
+            }
+        }
+
+
+        @Override
+        public VanillaBytes appendUtf8( CharSequence str) throws BufferOverflowException {
+            try {
+                if (isDirectMemory()) {
+                    if (str instanceof BytesStore) {
+                        write((BytesStore) str, 0L, str.length());
+                        return this;
+                    }
+                    if (str instanceof String) {
+                        if (Jvm.isJava9Plus()) {
+                            String str1 = (String) str;
+                            byte coder = StringUtils.getStringCoder(str1);
+                            appendUtf8(StringUtils.extractBytes(str1), 0, str.length(), coder);
+                        } else {
+                            appendUtf8(StringUtils.extractChars((String) str), 0, str.length());
+                        }
+                        return this;
+                    }
+                }
+                super.append(str, 0, str.length());
+                return this;
+
+            } catch (Exception e) {
+                 BufferOverflowException e2 = new BufferOverflowException();
+                e2.initCause(e);
+                throw e2;
+            }
+        }
+
+        @Override
+
+        public Bytes<Underlying> append8bit( CharSequence cs)
+                throws BufferOverflowException, BufferUnderflowException, IndexOutOfBoundsException {
+            if (cs instanceof RandomDataInput)
+                return write((RandomDataInput) cs);
+
+            if (isDirectMemory() && cs instanceof String)
+                return append8bitNBS_S((String) cs);
+            return append8bit0(cs);
+        }
+
+        @Override
+
+        public Bytes<Underlying> append8bit( BytesStore bs)
+                throws BufferOverflowException, BufferUnderflowException, IndexOutOfBoundsException {
+            long remaining = bs.readLimit() - bs.readPosition();
+            return write(bs, 0L, remaining);
+        }
+
+
+        @Override
+        public Bytes<Underlying> write( BytesStore bytes, long offset, long length) throws BufferOverflowException, BufferUnderflowException {
+            if (bytes.canReadDirect(length) && canWriteDirect(length) && length == (int) length) {
+                long wAddr = addressForWritePosition();
+                writeSkip(length);
+                long rAddr = bytes.addressForRead(offset);
+                BytesInternal.copyMemory(rAddr, wAddr, (int) length);
+
+            } else {
+                BytesInternal.writeFully(bytes, offset, length, this);
+            }
+            return this;
+        }
+
+        @Override
+
+        public Bytes<Underlying> append8bit( String cs)
+                throws BufferOverflowException, BufferUnderflowException, IndexOutOfBoundsException {
+            if (isDirectMemory())
+                return append8bitNBS_S(cs);
+            return append8bit0(cs);
+        }
+
+
+        private Bytes<Underlying> append8bitNBS_S( String s) throws BufferOverflowException {
+            int length = s.length();
+            long offset = writeOffsetPositionMoved(length); // can re-assign the byteStore if not large enough.
+             NativeBytesStore bytesStore = (NativeBytesStore) this.bytesStore;
+            final long address = bytesStore.address + bytesStore.translate(offset);
+             final Memory memory = bytesStore.memory;
+
+            if (memory == null)
+                bytesStore.throwExceptionIfReleased();
+
+            if (Jvm.isJava9Plus()) {
+                final byte[] chars = StringUtils.extractBytes(s);
+
+                int i;
+                for (i = 0; i < length; i++) {
+                    memory.writeByte(address + i, chars[i]);
+                }
+            } else {
+                final char[] chars = StringUtils.extractChars(s);
+                int i;
+                for (i = 0; i < length - 3; i += 4) {
+                    int c0 = chars[i] & 0xFF;
+                    int c1 = chars[i + 1] & 0xFF;
+                    int c2 = chars[i + 2] & 0xFF;
+                    int c3 = chars[i + 3] & 0xFF;
+                    memory.writeInt(address + i, c0 | (c1 << 8) | (c2 << 16) | (c3 << 24));
+                }
+                for (; i < length; i++) {
+                    int c0 = chars[i];
+                    memory.writeByte(address + i, (byte) c0);
+                }
+            }
+            return this;
+        }
+
+
+        public String toString() {
+            return bytesStore instanceof NativeBytesStore
+                    ? toString2((NativeBytesStore) bytesStore)
+                    : toString0();
+        }
+
+        private String toString2( NativeBytesStore bytesStore) {
+            int length = (int)
+                    Math.min(Bytes.MAX_HEAP_CAPACITY, readRemaining());
+             char[] chars = new char[length];
+             final Memory memory = bytesStore.memory;
+            final long address = bytesStore.address + bytesStore.translate(readPosition());
+            for (int i = 0; i < length && i < realCapacity(); i++)
+                chars[i] = (char) (memory.readByte(address + i) & 0xFF);
+
+            return StringUtils.newString(chars);
+        }
+
+
+        protected String toString0() {
+            int length = (int) Math.min(Bytes.MAX_HEAP_CAPACITY, readRemaining());
+             char[] chars = new char[length];
+            try {
+                for (int i = 0; i < length; i++) {
+                    chars[i] = (char) (bytesStore.readByte(readPosition() + i) & 0xFF);
+                }
+            } catch (BufferUnderflowException e) {
+                // ignored
+            }
+            return StringUtils.newString(chars);
+        }
+
+
+        protected Bytes<Underlying> append8bit0( CharSequence cs) throws BufferOverflowException, IndexOutOfBoundsException {
+            int length = cs.length();
+            long offset = writeOffsetPositionMoved(length);
+            for (int i = 0; i < length; i++) {
+                char c = cs.charAt(i);
+                if (c > 255) c = '?';
+                bytesStore.writeByte(offset + i, (byte) c);
+            }
+            return this;
+        }
+
+        @Override
+        public boolean equalBytes( BytesStore bytesStore, long length) throws BufferUnderflowException {
+            if (isDirectMemory() &&
+                    bytesStore instanceof VanillaBytes &&
+                    bytesStore.isDirectMemory()) {
+                 VanillaBytes b2 = (VanillaBytes) bytesStore;
+                 NativeBytesStore nbs0 = (NativeBytesStore) this.bytesStore;
+                 NativeBytesStore nbs2 = (NativeBytesStore) b2.bytesStore();
+                long i = 0;
+                for (; i < length - 7; i += 8) {
+                    long addr0 = nbs0.address + readPosition() - nbs0.start() + i;
+                    long addr2 = nbs2.address + b2.readPosition() - nbs2.start() + i;
+                    long l0 = nbs0.memory.readLong(addr0);
+                    long l2 = nbs2.memory.readLong(addr2);
+                    if (l0 != l2)
+                        return false;
+                }
+                for (; i < length; i++) {
+                    long offset2 = readPosition() + i - nbs0.start();
+                    long offset21 = b2.readPosition() + i - nbs2.start();
+                    byte b0 = nbs0.memory.readByte(nbs0.address + offset2);
+                    byte b1 = nbs2.memory.readByte(nbs2.address + offset21);
+                    if (b0 != b1)
+                        return false;
+                }
+                return true;
+            } else {
+                return super.equalBytes(bytesStore, length);
+            }
+        }
+
+        public void read8Bit(char[] chars, int length) {
+            if (isDirectMemory()) {
+                long position = readPosition();
+                 NativeBytesStore nbs = (NativeBytesStore) bytesStore();
+                nbs.read8bit(position, chars, length);
+            } else {
+                long pos = this.readPosition();
+                for (int i = 0; i < length; i++)
+                    chars[i] = (char) this.readUnsignedByte(pos + i);
+            }
+        }
+
+        // TODO: protected?
+        @Override
+        public int byteCheckSum(int start, int end) throws IORuntimeException {
+            byte b = 0;
+            // the below cast is safe as should only be called from net.openhft.chronicle.bytes.AbstractBytes.byteCheckSum(long, long)
+             NativeBytesStore bytesStore = (NativeBytesStore) bytesStore();
+             Memory memory = bytesStore.memory;
+            assert memory != null;
+            long addr = bytesStore.addressForRead(start);
+            int i = 0, len = end - start;
+            for (; i < len - 3; i += 4) {
+                b += memory.readByte(addr + i)
+                        + memory.readByte(addr + i + 1)
+                        + memory.readByte(addr + i + 2)
+                        + memory.readByte(addr + i + 3);
+            }
+            for (; i < len; i++) {
+                b += memory.readByte(addr + i);
+            }
+            return b & 0xFF;
+        }
+
+
+        @Override
+        public Bytes<Underlying> appendUtf8(char[] chars, int offset, int length) throws BufferOverflowException, IllegalArgumentException {
+            ensureCapacity(length);
+            if (bytesStore instanceof NativeBytesStore) {
+                 NativeBytesStore nbs = (NativeBytesStore) this.bytesStore;
+                long position = nbs.appendUtf8(writePosition(), chars, offset, length);
+                writePosition(position);
+            } else {
+                super.appendUtf8(chars, offset, length);
+            }
+            return this;
+        }
+
+        @Override
+        public ByteBuffer toTemporaryDirectByteBuffer() throws IllegalArgumentException {
+            if (isClear())
+                return bytesStore.toTemporaryDirectByteBuffer();
+            return super.toTemporaryDirectByteBuffer();
+        }
+
+        @Override
+        public int read( byte[] bytes) {
+            int len = (int) Math.min(bytes.length, readRemaining());
+            if (bytesStore instanceof NativeBytesStore) {
+                 NativeBytesStore nbs = (NativeBytesStore) this.bytesStore;
+                long len2 = nbs.read(readPosition(), bytes, 0, len);
+                try {
+                    readSkip(len2);
+                } catch (BufferUnderflowException e) {
+                    throw new AssertionError(e);
+                }
+                return (int) (len2);
+            }
+            return super.read(bytes);
+        }
+
+        @Override
+        public int compareTo( CharSequence cs) {
+            long len1 = readRemaining();
+            int len2 = cs.length();
+            long lim = Math.min(len1, len2);
+
+            int k = 0;
+            while (k < lim) {
+                char c1 = bytesStore.charAt(k);
+                char c2 = cs.charAt(k);
+                if (c1 != c2) {
+                    return c1 - c2;
+                }
+                k++;
+            }
+            return (int) (len1 - len2);
+        }
+
+
+
+    }

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/WriteBytesMarshallable.java

@@ -0,0 +1,36 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes;
+
+import net.openhft.chronicle.core.annotation.DontChain;
+
+/**
+ * Write data directly as Bytes.
+ */
+@SuppressWarnings("rawtypes")
+@FunctionalInterface
+@DontChain
+public interface WriteBytesMarshallable extends CommonMarshallable {
+    /**
+     * Write to Bytes.  This can be used as an interface to extend or a lambda
+     *
+     * @param bytes to write to.
+     */
+    void writeMarshallable(BytesOut bytes);
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/algo/BytesStoreHash.java

@@ -0,0 +1,54 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes.algo;
+
+import net.openhft.chronicle.bytes.BytesStore;
+import net.openhft.chronicle.bytes.NativeBytesStore;
+
+import java.util.function.ToLongFunction;
+
+/**
+ * Simple function to derive a long hash from a BytesStore
+ */
+@SuppressWarnings("rawtypes")
+public interface BytesStoreHash<B extends BytesStore> extends ToLongFunction<B> {
+    static long hash( BytesStore b) {
+        return b.isDirectMemory() && b.bytesStore() instanceof NativeBytesStore
+                ? OptimisedBytesStoreHash.INSTANCE.applyAsLong(b)
+                : VanillaBytesStoreHash.INSTANCE.applyAsLong(b);
+    }
+
+    static long hash( BytesStore b, long length) {
+        return b.isDirectMemory() && b.bytesStore() instanceof NativeBytesStore
+                ? OptimisedBytesStoreHash.INSTANCE.applyAsLong(b, length)
+                : VanillaBytesStoreHash.INSTANCE.applyAsLong(b, length);
+    }
+
+    static int hash32(BytesStore b) {
+        long hash = hash(b);
+        return (int) (hash ^ (hash >>> 32));
+    }
+
+    static int hash32( BytesStore b, int length) {
+        long hash = hash(b, length);
+        return (int) (hash ^ (hash >>> 32));
+    }
+
+    long applyAsLong(BytesStore bytes, long length);
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/algo/OptimisedBytesStoreHash.java

@@ -0,0 +1,274 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes.algo;
+
+import net.openhft.chronicle.bytes.BytesStore;
+import net.openhft.chronicle.bytes.NativeBytesStore;
+import net.openhft.chronicle.core.Maths;
+import net.openhft.chronicle.core.Memory;
+import net.openhft.chronicle.core.OS;
+
+import java.nio.ByteOrder;
+
+import static net.openhft.chronicle.bytes.algo.VanillaBytesStoreHash.*;
+
+@SuppressWarnings("rawtypes")
+public enum OptimisedBytesStoreHash implements BytesStoreHash<BytesStore> {
+    INSTANCE;
+
+
+    public static final Memory MEMORY = OS.memory();
+    public static final boolean IS_LITTLE_ENDIAN = ByteOrder.nativeOrder() == ByteOrder.LITTLE_ENDIAN;
+    private static final int TOP_BYTES = IS_LITTLE_ENDIAN ? 4 : 0;
+
+    static long applyAsLong1to7( BytesStore store, int remaining) {
+        final long address = store.addressForRead(store.readPosition());
+
+        return hash(readIncompleteLong(address, remaining));
+    }
+
+    static long applyAsLong8( BytesStore store) {
+        final long address = store.addressForRead(store.readPosition());
+
+        return hash0(MEMORY.readLong(address), MEMORY.readInt(address + TOP_BYTES));
+    }
+
+    public static long hash(long l) {
+        return hash0(l, l >> 32);
+    }
+
+    static long hash0(long l, long hi) {
+        return agitate(l * K0 + hi * K1);
+    }
+
+    static long applyAsLong9to16( BytesStore store, int remaining) {
+         final NativeBytesStore bytesStore = (NativeBytesStore) store.bytesStore();
+        final long address = bytesStore.addressForRead(store.readPosition());
+        long h0 = (long) remaining * K0;
+
+        int left = remaining;
+        long addrI = address;
+
+        long l0 = readIncompleteLong(addrI, left);
+        int l0a = (int) (l0 >> 32);
+        long l1 = readIncompleteLong(addrI + 8, left - 8);
+        int l1a = (int) (l1 >> 32);
+        final long l2 = 0;
+        final int l2a = 0;
+        final long l3 = 0;
+        final int l3a = 0;
+
+        h0 += (l0 + l1a - l2a) * M0;
+        long h1 = (l1 + l2a - l3a) * M1;
+        long h2 = (l2 + l3a - l0a) * M2;
+        long h3 = (l3 + l0a - l1a) * M3;
+
+        return agitate(h0) ^ agitate(h1)
+                ^ agitate(h2) ^ agitate(h3);
+    }
+
+    static long applyAsLong17to32( BytesStore store, int remaining) {
+         final NativeBytesStore bytesStore = (NativeBytesStore) store.bytesStore();
+        final long address = bytesStore.addressForRead(store.readPosition());
+        long h0 = (long) remaining * K0;
+
+        int left = remaining;
+        long addrI = address;
+
+        long l0 = MEMORY.readLong(addrI);
+        int l0a = MEMORY.readInt(addrI + TOP_BYTES);
+        long l1 = MEMORY.readLong(addrI + 8);
+        int l1a = MEMORY.readInt(addrI + 8 + TOP_BYTES);
+        long l2 = readIncompleteLong(addrI + 16, left - 16);
+        int l2a = (int) (l2 >> 32);
+        long l3 = readIncompleteLong(addrI + 24, left - 24);
+        int l3a = (int) (l3 >> 32);
+
+        h0 += (l0 + l1a - l2a) * M0;
+        long h1 = (l1 + l2a - l3a) * M1;
+        long h2 = (l2 + l3a - l0a) * M2;
+        long h3 = (l3 + l0a - l1a) * M3;
+
+        return agitate(h0) ^ agitate(h1)
+                ^ agitate(h2) ^ agitate(h3);
+    }
+
+    public static long applyAsLong32bytesMultiple( BytesStore store, int remaining) {
+         final NativeBytesStore bytesStore = (NativeBytesStore) store.bytesStore();
+        final long address = bytesStore.addressForRead(store.readPosition());
+        long h0 = remaining * K0, h1 = 0, h2 = 0, h3 = 0;
+
+        int i;
+        for (i = 0; i < remaining - 31; i += 32) {
+            if (i > 0) {
+                h0 *= K0;
+                h1 *= K1;
+                h2 *= K2;
+                h3 *= K3;
+            }
+            long addrI = address + i;
+            long l0 = MEMORY.readLong(addrI);
+            int l0a = MEMORY.readInt(addrI + TOP_BYTES);
+            long l1 = MEMORY.readLong(addrI + 8);
+            int l1a = MEMORY.readInt(addrI + 8 + TOP_BYTES);
+            long l2 = MEMORY.readLong(addrI + 16);
+            int l2a = MEMORY.readInt(addrI + 16 + TOP_BYTES);
+            long l3 = MEMORY.readLong(addrI + 24);
+            int l3a = MEMORY.readInt(addrI + 24 + TOP_BYTES);
+
+            h0 += (l0 + l1a - l2a) * M0;
+            h1 += (l1 + l2a - l3a) * M1;
+            h2 += (l2 + l3a - l0a) * M2;
+            h3 += (l3 + l0a - l1a) * M3;
+        }
+
+        return agitate(h0) ^ agitate(h1)
+                ^ agitate(h2) ^ agitate(h3);
+    }
+
+    public static long applyAsLongAny( BytesStore store, long remaining) {
+         final NativeBytesStore bytesStore = (NativeBytesStore) store.bytesStore();
+        final long address = bytesStore.addressForRead(store.readPosition());
+        long h0 = remaining * K0, h1 = 0, h2 = 0, h3 = 0;
+
+        int i;
+        for (i = 0; i < remaining - 31; i += 32) {
+            if (i > 0) {
+                h0 *= K0;
+                h1 *= K1;
+                h2 *= K2;
+                h3 *= K3;
+            }
+            long addrI = address + i;
+            long l0 = MEMORY.readLong(addrI);
+            int l0a = MEMORY.readInt(addrI + TOP_BYTES);
+            long l1 = MEMORY.readLong(addrI + 8);
+            int l1a = MEMORY.readInt(addrI + 8 + TOP_BYTES);
+            long l2 = MEMORY.readLong(addrI + 16);
+            int l2a = MEMORY.readInt(addrI + 16 + TOP_BYTES);
+            long l3 = MEMORY.readLong(addrI + 24);
+            int l3a = MEMORY.readInt(addrI + 24 + TOP_BYTES);
+
+            h0 += (l0 + l1a - l2a) * M0;
+            h1 += (l1 + l2a - l3a) * M1;
+            h2 += (l2 + l3a - l0a) * M2;
+            h3 += (l3 + l0a - l1a) * M3;
+        }
+        long left = remaining - i;
+        if (left > 0) {
+            if (i > 0) {
+                h0 *= K0;
+                h1 *= K1;
+                h2 *= K2;
+                h3 *= K3;
+            }
+            long addrI = address + i;
+            if (left <= 16) {
+
+                long l0 = readIncompleteLong(addrI, (int) left);
+                int l0a = (int) (l0 >> 32);
+                long l1 = readIncompleteLong(addrI + 8, (int) (left - 8));
+                int l1a = (int) (l1 >> 32);
+                final long l2 = 0;
+                final int l2a = 0;
+                final long l3 = 0;
+                final int l3a = 0;
+
+                h0 += (l0 + l1a - l2a) * M0;
+                h1 += (l1 + l2a - l3a) * M1;
+                h2 += (l2 + l3a - l0a) * M2;
+                h3 += (l3 + l0a - l1a) * M3;
+
+            } else {
+                long l0 = MEMORY.readLong(addrI);
+                int l0a = MEMORY.readInt(addrI + TOP_BYTES);
+                long l1 = MEMORY.readLong(addrI + 8);
+                int l1a = MEMORY.readInt(addrI + 8 + TOP_BYTES);
+                long l2 = readIncompleteLong(addrI + 16, (int) (left - 16));
+                int l2a = (int) (l2 >> 32);
+                long l3 = readIncompleteLong(addrI + 24, (int) (left - 24));
+                int l3a = (int) (l3 >> 32);
+
+                h0 += (l0 + l1a - l2a) * M0;
+                h1 += (l1 + l2a - l3a) * M1;
+                h2 += (l2 + l3a - l0a) * M2;
+                h3 += (l3 + l0a - l1a) * M3;
+            }
+        }
+
+        return agitate(h0) ^ agitate(h1)
+                ^ agitate(h2) ^ agitate(h3);
+    }
+
+    static long readIncompleteLong(long address, int len) {
+        switch (len) {
+            case 1:
+                return MEMORY.readByte(address);
+            case 2:
+                return MEMORY.readShort(address);
+            case 3:
+                return IS_LITTLE_ENDIAN
+                        ? (MEMORY.readShort(address) & 0xFFFF) + ((MEMORY.readByte(address + 2) & 0xFF) << 16)
+                        : ((MEMORY.readShort(address) & 0xFFFF) << 8) + (MEMORY.readByte(address + 2) & 0xFF);
+            case 4:
+                return MEMORY.readInt(address);
+            case 5:
+                return IS_LITTLE_ENDIAN
+                        ? (MEMORY.readInt(address) & 0xFFFFFFFFL) + ((long) (MEMORY.readByte(address + 4) & 0xFF) << 32)
+                        : ((MEMORY.readInt(address) & 0xFFFFFFFFL) << 8) + (MEMORY.readByte(address + 4) & 0xFF);
+            case 6:
+                return IS_LITTLE_ENDIAN
+                        ? (MEMORY.readInt(address) & 0xFFFFFFFFL) + ((long) (MEMORY.readShort(address + 4) & 0xFFFF) << 32)
+                        : ((MEMORY.readInt(address) & 0xFFFFFFFFL) << 16) + (MEMORY.readShort(address + 4) & 0xFFFF);
+            case 7:
+                return IS_LITTLE_ENDIAN
+                        ? (MEMORY.readInt(address) & 0xFFFFFFFFL) + ((long) (MEMORY.readShort(address + 4) & 0xFFFF) << 32) + ((long) (MEMORY.readByte(address + 6) & 0xFF) << 48)
+                        : ((MEMORY.readInt(address) & 0xFFFFFFFFL) << 24) + ((MEMORY.readShort(address + 4) & 0xFFFF) << 8) + (MEMORY.readByte(address + 6) & 0xFF);
+            default:
+                return len >= 8 ? MEMORY.readLong(address) : 0;
+        }
+    }
+
+    @Override
+    public long applyAsLong( BytesStore store) {
+        final int remaining = Maths.toInt32(store.readRemaining());
+        return applyAsLong(store, remaining);
+    }
+
+    @Override
+    public long applyAsLong( BytesStore store, long remaining) {
+        if (remaining <= 16) {
+            if (remaining == 0) {
+                return 0;
+            } else if (remaining < 8) {
+                return applyAsLong1to7(store, (int) remaining);
+            } else if (remaining == 8) {
+                return applyAsLong8(store);
+            } else {
+                return applyAsLong9to16(store, (int) remaining);
+            }
+        } else if (remaining <= 32) {
+            return applyAsLong17to32(store, (int) remaining);
+        } else if ((remaining & 31) == 0) {
+            return applyAsLong32bytesMultiple(store, (int) remaining);
+        } else {
+            return applyAsLongAny(store, remaining);
+        }
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/algo/VanillaBytesStoreHash.java

@@ -0,0 +1,116 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes.algo;
+
+import net.openhft.chronicle.bytes.BytesStore;
+import net.openhft.chronicle.core.Maths;
+import org.jetbrains.annotations.NotNull;
+
+import java.nio.ByteOrder;
+
+@SuppressWarnings("rawtypes")
+public enum VanillaBytesStoreHash implements BytesStoreHash<BytesStore> {
+    INSTANCE;
+
+    public static final int K0 = 0x6d0f27bd;
+    public static final int K1 = 0xc1f3bfc9;
+    public static final int K2 = 0x6b192397;
+    public static final int K3 = 0x6b915657;
+    public static final int M0 = 0x5bc80bad;
+    public static final int M1 = 0xea7585d7;
+    public static final int M2 = 0x7a646e19;
+    public static final int M3 = 0x855dd4db;
+    private static final int HI_BYTES = ByteOrder.nativeOrder() == ByteOrder.LITTLE_ENDIAN ? 4 : 0;
+
+    public static long agitate(long l) {
+        l ^= Long.rotateLeft(l, 26);
+        l ^= Long.rotateRight(l, 17);
+        return l;
+    }
+
+    @Override
+    public long applyAsLong( BytesStore store) {
+        int remaining = Maths.toInt32(store.readRemaining());
+        return applyAsLong(store, remaining);
+    }
+
+    @Override
+    public long applyAsLong(BytesStore bytes, long length) {
+        long start = bytes.readPosition();
+        if (length <= 8) {
+            long l = bytes.readIncompleteLong(start);
+            return agitate(l * K0 + (l >> 32) * K1);
+        }
+        // use two hashes so that when they are combined the 64-bit hash is more random.
+        long h0 = length * K0;
+        long h1 = 0, h2 = 0, h3 = 0;
+        int i;
+        // optimise chunks of 32 bytes but this is the same as the next loop.
+        for (i = 0; i < length - 31; i += 32) {
+            if (i > 0) {
+                h0 *= K0;
+                h1 *= K1;
+                h2 *= K2;
+                h3 *= K3;
+            }
+            long addrI = start + i;
+            long l0 = bytes.readLong(addrI);
+            int l0a = bytes.readInt(addrI + HI_BYTES);
+            long l1 = bytes.readLong(addrI + 8);
+            int l1a = bytes.readInt(addrI + 8 + HI_BYTES);
+            long l2 = bytes.readLong(addrI + 16);
+            int l2a = bytes.readInt(addrI + 16 + HI_BYTES);
+            long l3 = bytes.readLong(addrI + 24);
+            int l3a = bytes.readInt(addrI + 24 + HI_BYTES);
+
+            h0 += (l0 + l1a - l2a) * M0;
+            h1 += (l1 + l2a - l3a) * M1;
+            h2 += (l2 + l3a - l0a) * M2;
+            h3 += (l3 + l0a - l1a) * M3;
+        }
+
+        // perform a hash of the end.
+        long left = length - i;
+        if (left > 0) {
+            if (i > 0) {
+                h0 *= K0;
+                h1 *= K1;
+                h2 *= K2;
+                h3 *= K3;
+            }
+
+            long addrI = start + i;
+            long l0 = bytes.readIncompleteLong(addrI);
+            int l0a = (int) (l0 >> 32);
+            long l1 = bytes.readIncompleteLong(addrI + 8);
+            int l1a = (int) (l1 >> 32);
+            long l2 = bytes.readIncompleteLong(addrI + 16);
+            int l2a = (int) (l2 >> 32);
+            long l3 = bytes.readIncompleteLong(addrI + 24);
+            int l3a = (int) (l3 >> 32);
+
+            h0 += (l0 + l1a - l2a) * M0;
+            h1 += (l1 + l2a - l3a) * M1;
+            h2 += (l2 + l3a - l0a) * M2;
+            h3 += (l3 + l0a - l1a) * M3;
+        }
+        return agitate(h0) ^ agitate(h1)
+                ^ agitate(h2) ^ agitate(h3);
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/algo/XxHash.java

@@ -0,0 +1,161 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes.algo;
+
+import net.openhft.chronicle.bytes.BytesStore;
+
+// Migration of XxHash from Zero-Allocation-Hashing
+@SuppressWarnings("rawtypes")
+public class XxHash implements BytesStoreHash<BytesStore> {
+    // Primes if treated as unsigned
+    private static final long P1 = -7046029288634856825L;
+    private static final long P2 = -4417276706812531889L;
+    private static final long P3 = 1609587929392839161L;
+    private static final long P4 = -8796714831421723037L;
+    public static final XxHash INSTANCE = new XxHash(P4);
+    private static final long P5 = 2870177450012600261L;
+    private final long seed;
+
+    public XxHash(long seed) {
+        this.seed = seed;
+    }
+
+    private static long finalize(long hash) {
+        hash ^= hash >>> 33;
+        hash *= P2;
+        hash ^= hash >>> 29;
+        hash *= P3;
+        hash ^= hash >>> 32;
+        return hash;
+    }
+
+    long fetch64(BytesStore bytes, long off) {
+        return bytes.readLong(off);
+    }
+
+    // long because of unsigned nature of original algorithm
+    long fetch32(BytesStore bytes, long off) {
+        return bytes.readUnsignedInt(off);
+    }
+
+    // int because of unsigned nature of original algorithm
+    long fetch8(BytesStore bytes, long off) {
+        return bytes.readUnsignedByte(off);
+    }
+
+    @Override
+    public long applyAsLong(BytesStore bytes) {
+        return applyAsLong(bytes, bytes.readRemaining());
+    }
+
+    @Override
+    public long applyAsLong(BytesStore bytes, long length) {
+        long hash;
+        long remaining = length;
+        long off = bytes.readPosition();
+
+        if (remaining >= 32) {
+            long v1 = seed + P1 + P2;
+            long v2 = seed + P2;
+            long v3 = seed;
+            long v4 = seed - P1;
+
+            do {
+                v1 += fetch64(bytes, off) * P2;
+                v1 = Long.rotateLeft(v1, 31);
+                v1 *= P1;
+
+                v2 += fetch64(bytes, off + 8) * P2;
+                v2 = Long.rotateLeft(v2, 31);
+                v2 *= P1;
+
+                v3 += fetch64(bytes, off + 16) * P2;
+                v3 = Long.rotateLeft(v3, 31);
+                v3 *= P1;
+
+                v4 += fetch64(bytes, off + 24) * P2;
+                v4 = Long.rotateLeft(v4, 31);
+                v4 *= P1;
+
+                off += 32;
+                remaining -= 32;
+            } while (remaining >= 32);
+
+            hash = Long.rotateLeft(v1, 1)
+                    + Long.rotateLeft(v2, 7)
+                    + Long.rotateLeft(v3, 12)
+                    + Long.rotateLeft(v4, 18);
+
+            v1 *= P2;
+            v1 = Long.rotateLeft(v1, 31);
+            v1 *= P1;
+            hash ^= v1;
+            hash = hash * P1 + P4;
+
+            v2 *= P2;
+            v2 = Long.rotateLeft(v2, 31);
+            v2 *= P1;
+            hash ^= v2;
+            hash = hash * P1 + P4;
+
+            v3 *= P2;
+            v3 = Long.rotateLeft(v3, 31);
+            v3 *= P1;
+            hash ^= v3;
+            hash = hash * P1 + P4;
+
+            v4 *= P2;
+            v4 = Long.rotateLeft(v4, 31);
+            v4 *= P1;
+            hash ^= v4;
+            hash = hash * P1 + P4;
+        } else {
+            hash = seed + P5;
+        }
+
+        hash += length;
+
+        while (remaining >= 8) {
+            long k1 = fetch64(bytes, off);
+            k1 *= P2;
+            k1 = Long.rotateLeft(k1, 31);
+            k1 *= P1;
+            hash ^= k1;
+            hash = Long.rotateLeft(hash, 27) * P1 + P4;
+            off += 8;
+            remaining -= 8;
+        }
+
+        if (remaining >= 4) {
+            hash ^= fetch32(bytes, off) * P1;
+            hash = Long.rotateLeft(hash, 23) * P2 + P3;
+            off += 4;
+            remaining -= 4;
+        }
+
+        while (remaining != 0) {
+            hash ^= fetch8(bytes, off) * P5;
+            hash = Long.rotateLeft(hash, 11) * P1;
+            --remaining;
+            ++off;
+        }
+
+        return finalize(hash);
+    }
+
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/internal/package-info.java

@@ -0,0 +1,16 @@
+/**
+ * This package and any and all sub-packages contains strictly internal classes for this Chronicle library.
+ * Internal classes shall <em>never</em> be used directly.
+ * <p>
+ *  Specifically, the following actions (including, but not limited to) are not allowed
+ *  on internal classes and packages:
+ *  <ul>
+ *      <li>Casting to</li>
+ *      <li>Reflection of any kind</li>
+ *      <li>Explicit Serialize/deserialize</li>
+ *  </ul>
+ * <p>
+ * The classes in this package and any sub-package are subject to
+ * changes at any time for any reason.
+ */
+package net.openhft.chronicle.bytes.internal;

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/pool/BytesPool.java

@@ -0,0 +1,48 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package net.openhft.chronicle.bytes.pool;
+
+import net.openhft.chronicle.bytes.Bytes;
+import net.openhft.chronicle.core.io.IOTools;
+import org.jetbrains.annotations.NotNull;
+
+import java.nio.ByteBuffer;
+
+@SuppressWarnings("rawtypes")
+public class BytesPool {
+    final ThreadLocal<Bytes> bytesTL = new ThreadLocal<>();
+
+    public Bytes acquireBytes() {
+        Bytes bytes = bytesTL.get();
+        if (bytes == null) {
+            bytesTL.set(bytes = createBytes());
+        } else {
+            bytes.clear();
+        }
+        return bytes;
+    }
+
+
+    protected Bytes createBytes() {
+        // use heap buffer as we never know when a thread will die and not release this resource.
+        Bytes<ByteBuffer> bbb = Bytes.elasticHeapByteBuffer(256);
+        IOTools.unmonitor(bbb);
+        return bbb;
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/ref/AbstractReference.java

@@ -0,0 +1,93 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes.ref;
+
+import net.openhft.chronicle.bytes.Byteable;
+import net.openhft.chronicle.bytes.BytesStore;
+import net.openhft.chronicle.core.io.AbstractCloseable;
+import net.openhft.chronicle.core.io.Closeable;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+import java.nio.BufferOverflowException;
+import java.nio.BufferUnderflowException;
+
+@SuppressWarnings("rawtypes")
+public abstract class AbstractReference extends AbstractCloseable implements Byteable, Closeable {
+
+
+    protected BytesStore<?, ?> bytes;
+    protected long offset;
+
+    @Override
+    public void bytesStore( final BytesStore bytes, final long offset, final long length) throws IllegalStateException, IllegalArgumentException, BufferOverflowException, BufferUnderflowException {
+        throwExceptionIfClosedInSetter();
+
+        acceptNewBytesStore(bytes);
+        this.offset = offset;
+    }
+
+
+    @Override
+    public BytesStore bytesStore() {
+        return bytes;
+    }
+
+    @Override
+    public long offset() {
+        return offset;
+    }
+
+    @Override
+    public abstract long maxSize();
+
+    protected void acceptNewBytesStore( final BytesStore bytes) {
+        if (this.bytes != null) {
+            this.bytes.release(this);
+        }
+        this.bytes = bytes.bytesStore();
+        this.bytes.reserve(this);
+    }
+
+    @Override
+    protected void performClose() {
+        if (this.bytes != null) {
+            this.bytes.release(this);
+            this.bytes = null;
+        }
+    }
+
+    public long address() {
+        throwExceptionIfClosed();
+
+        return bytesStore().addressForRead(offset);
+    }
+
+/* TODO FIX
+        @Override
+    protected void finalize() throws Throwable {
+        warnIfNotClosed();
+        super.finalize();
+    }*/
+
+    @Override
+    protected boolean threadSafetyCheck(boolean isUsed) {
+        // References are thread safe
+        return true;
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/ref/BinaryBooleanReference.java

@@ -0,0 +1,51 @@
+package net.openhft.chronicle.bytes.ref;
+
+import net.openhft.chronicle.bytes.BytesStore;
+import net.openhft.chronicle.core.values.BooleanValue;
+import org.jetbrains.annotations.NotNull;
+
+import java.nio.BufferOverflowException;
+import java.nio.BufferUnderflowException;
+
+public class BinaryBooleanReference extends AbstractReference implements BooleanValue {
+
+    private static final byte FALSE = (byte) 0xB0;
+    private static final byte TRUE = (byte) 0xB1;
+
+    @SuppressWarnings("rawtypes")
+    @Override
+    public void bytesStore( final BytesStore bytes, final long offset, final long length) throws IllegalStateException, IllegalArgumentException, BufferOverflowException, BufferUnderflowException {
+        throwExceptionIfClosedInSetter();
+
+        if (length != maxSize())
+            throw new IllegalArgumentException();
+
+        super.bytesStore(bytes, offset, length);
+    }
+
+    @Override
+    public long maxSize() {
+        return 1;
+    }
+
+    @Override
+    public boolean getValue() {
+        throwExceptionIfClosed();
+
+        byte b = bytes.readByte(offset);
+        if (b == FALSE)
+            return false;
+        if (b == TRUE)
+            return true;
+
+        throw new IllegalStateException("unexpected code=" + b);
+
+    }
+
+    @Override
+    public void setValue(final boolean flag) {
+        throwExceptionIfClosed();
+
+        bytes.writeByte(offset, flag ? TRUE : FALSE);
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/ref/BinaryIntArrayReference.java

@@ -0,0 +1,282 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes.ref;
+
+import net.openhft.chronicle.bytes.*;
+import net.openhft.chronicle.core.io.IORuntimeException;
+import net.openhft.chronicle.core.values.IntValue;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+import java.lang.ref.WeakReference;
+import java.nio.BufferOverflowException;
+import java.nio.BufferUnderflowException;
+import java.util.Collections;
+import java.util.IdentityHashMap;
+import java.util.Set;
+
+import static net.openhft.chronicle.bytes.ref.BinaryIntReference.INT_NOT_COMPLETE;
+
+/**
+ * This class acts a Binary array of 64-bit values. c.f. TextLongArrayReference
+ */
+@SuppressWarnings("rawtypes")
+public class BinaryIntArrayReference extends AbstractReference implements ByteableIntArrayValues, BytesMarshallable {
+    public static final int SHIFT = 2;
+    private static final long CAPACITY = 0;
+    private static final long USED = CAPACITY + Long.BYTES;
+    private static final long VALUES = USED + Long.BYTES;
+    private static final int MAX_TO_STRING = 1024;
+
+    private static Set<WeakReference<BinaryIntArrayReference>> binaryIntArrayReferences = null;
+    private long length;
+
+    public BinaryIntArrayReference() {
+        this(0);
+    }
+
+    public BinaryIntArrayReference(long defaultCapacity) {
+        this.length = (defaultCapacity << SHIFT) + VALUES;
+    }
+
+    public static void startCollecting() {
+        binaryIntArrayReferences = Collections.newSetFromMap(new IdentityHashMap<>());
+    }
+
+    public static void forceAllToNotCompleteState() {
+        binaryIntArrayReferences.forEach(x -> {
+             BinaryIntArrayReference binaryLongReference = x.get();
+            if (binaryLongReference != null) {
+                binaryLongReference.setValueAt(0, INT_NOT_COMPLETE);
+            }
+        });
+        binaryIntArrayReferences = null;
+    }
+
+    public static void write( Bytes bytes, long capacity) throws BufferOverflowException, IllegalArgumentException {
+        assert (bytes.writePosition() & 0x7) == 0;
+
+        bytes.writeLong(capacity);
+        bytes.writeLong(0L); // used
+        long start = bytes.writePosition();
+        bytes.zeroOut(start, start + (capacity << SHIFT));
+        bytes.writeSkip(capacity << SHIFT);
+    }
+
+    public static void lazyWrite( Bytes bytes, long capacity) throws BufferOverflowException {
+        assert (bytes.writePosition() & 0x7) == 0;
+
+        bytes.writeLong(capacity);
+        bytes.writeLong(0L); // used
+        bytes.writeSkip(capacity << SHIFT);
+    }
+
+    public static long peakLength( BytesStore bytes, long offset) throws BufferUnderflowException {
+        final long capacity = bytes.readLong(offset + CAPACITY);
+        assert capacity > 0 : "capacity too small " + capacity;
+        return (capacity << SHIFT) + VALUES;
+    }
+
+    @Override
+    public long getCapacity() {
+        throwExceptionIfClosed();
+
+        return (length - VALUES) >>> SHIFT;
+    }
+
+    @Override
+    public long getUsed() {
+        throwExceptionIfClosed();
+
+        return bytes.readVolatileInt(offset + USED);
+    }
+
+    @Override
+    public void setMaxUsed(long usedAtLeast) {
+        throwExceptionIfClosedInSetter();
+
+        bytes.writeMaxLong(offset + USED, usedAtLeast);
+    }
+
+    @Override
+    public int getValueAt(long index) throws BufferUnderflowException {
+        throwExceptionIfClosed();
+
+        return bytes.readInt(VALUES + offset + (index << SHIFT));
+    }
+
+    @Override
+    public void setValueAt(long index, int value) throws IllegalArgumentException, BufferOverflowException {
+        throwExceptionIfClosedInSetter();
+
+        bytes.writeInt(VALUES + offset + (index << SHIFT), value);
+    }
+
+    @Override
+    public int getVolatileValueAt(long index) throws BufferUnderflowException {
+        throwExceptionIfClosed();
+
+        return bytes.readVolatileInt(VALUES + offset + (index << SHIFT));
+    }
+
+    @Override
+    public void bindValueAt(long index,  IntValue value) {
+        throwExceptionIfClosed();
+
+        ((BinaryIntReference) value).bytesStore(bytes, VALUES + offset + (index << SHIFT), 8);
+    }
+
+    @Override
+    public void setOrderedValueAt(long index, int value) throws IllegalArgumentException, BufferOverflowException {
+        throwExceptionIfClosedInSetter();
+
+        bytes.writeOrderedInt(VALUES + offset + (index << SHIFT), value);
+    }
+
+    @Override
+    public void bytesStore( BytesStore bytes, long offset, long length) throws BufferUnderflowException, IllegalArgumentException {
+        throwExceptionIfClosed();
+
+        if (length != peakLength(bytes, offset))
+            throw new IllegalArgumentException(length + " != " + peakLength(bytes, offset));
+
+        assert (offset & 7) == 0 : "offset=" + offset;
+        super.bytesStore(bytes, (offset + 7) & ~7, length);
+        this.length = length;
+    }
+
+    @Override
+    public void readMarshallable(BytesIn bytes) throws IORuntimeException {
+        throwExceptionIfClosedInSetter();
+
+        long position = bytes.readPosition();
+        long capacity = bytes.readLong();
+        long used = bytes.readLong();
+        if (capacity < 0 || capacity > bytes.readRemaining() >> SHIFT)
+            throw new IORuntimeException("Corrupt used capacity");
+
+        if (used < 0 || used > capacity)
+            throw new IORuntimeException("Corrupt used value");
+
+        bytes.readSkip(capacity << SHIFT);
+        long length = bytes.readPosition() - position;
+        bytesStore(((Bytes) bytes).bytesStore(), position, length);
+    }
+
+    @Override
+    public void writeMarshallable(BytesOut bytes) {
+        BytesStore bytesStore = bytesStore();
+        if (bytesStore == null) {
+            long capacity = getCapacity();
+            bytes.writeLong(capacity);
+            bytes.writeLong(0);
+            bytes.writeSkip(capacity << SHIFT);
+        } else {
+            bytes.write(bytesStore, offset, length);
+        }
+    }
+
+    @Override
+    public boolean isNull() {
+        throwExceptionIfClosed();
+
+        return bytes == null;
+    }
+
+    @Override
+    public void reset() {
+        throwExceptionIfClosedInSetter();
+
+        bytes = null;
+        offset = 0;
+        length = 0;
+    }
+
+
+    @Override
+    public BytesStore bytesStore() {
+        return bytes;
+    }
+
+    @Override
+    public long offset() {
+        return offset;
+    }
+
+    @Override
+    public long maxSize() {
+        return length;
+    }
+
+
+    public String toString() {
+        if (bytes == null)
+            return "not set";
+         StringBuilder sb = new StringBuilder();
+        sb.append("used: ");
+        long used = getUsed();
+        sb.append(used);
+        sb.append(", value: ");
+         String sep = "";
+        try {
+            int i, max = (int) Math.min(used, Math.min(getCapacity(), MAX_TO_STRING));
+            for (i = 0; i < max; i++) {
+                long valueAt = getValueAt(i);
+                sb.append(sep).append(valueAt);
+                sep = ", ";
+            }
+            if (i < getCapacity())
+                sb.append(" ...");
+
+        } catch (BufferUnderflowException e) {
+            sb.append(" ").append(e);
+        }
+        return sb.toString();
+    }
+
+    @Override
+    public long sizeInBytes(long capacity) {
+        throwExceptionIfClosed();
+
+        return (capacity << SHIFT) + VALUES;
+    }
+
+    @Override
+    public ByteableIntArrayValues capacity(long arrayLength) {
+        throwExceptionIfClosedInSetter();
+
+        BytesStore bytesStore = bytesStore();
+        long length = sizeInBytes(arrayLength);
+        if (bytesStore == null) {
+            this.length = length;
+        } else {
+            assert this.length == length;
+        }
+        return this;
+    }
+
+    @Override
+    public boolean compareAndSet(long index, int expected, int value) throws IllegalArgumentException, BufferOverflowException {
+        throwExceptionIfClosed();
+
+        if (value == INT_NOT_COMPLETE && binaryIntArrayReferences != null)
+            binaryIntArrayReferences.add(new WeakReference<>(this));
+        return bytes.compareAndSwapInt(VALUES + offset + (index << SHIFT), expected, value);
+    }
+}
+

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/ref/BinaryIntReference.java

@@ -0,0 +1,101 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes.ref;
+
+import net.openhft.chronicle.bytes.BytesStore;
+import net.openhft.chronicle.core.values.IntValue;
+import org.jetbrains.annotations.NotNull;
+
+import java.nio.BufferOverflowException;
+import java.nio.BufferUnderflowException;
+
+/**
+ * This class acts as a Binary 32-bit in values. c.f. TextIntReference
+ */
+public class BinaryIntReference extends AbstractReference implements IntValue {
+    public static final int INT_NOT_COMPLETE = Integer.MIN_VALUE;
+
+    @SuppressWarnings("rawtypes")
+    @Override
+    public void bytesStore( final BytesStore bytes, final long offset, final long length) throws IllegalStateException, IllegalArgumentException, BufferOverflowException, BufferUnderflowException {
+        throwExceptionIfClosedInSetter();
+
+        if (length != maxSize())
+            throw new IllegalArgumentException();
+        super.bytesStore(bytes, offset, length);
+    }
+
+    @Override
+    public long maxSize() {
+        return 4;
+    }
+
+
+    public String toString() {
+        return bytes == null ? "bytes is null" : "value: " + getValue();
+    }
+
+    @Override
+    public int getValue() {
+        throwExceptionIfClosed();
+
+        return bytes == null ? 0 : bytes.readInt(offset);
+    }
+
+    @Override
+    public void setValue(int value) {
+        throwExceptionIfClosedInSetter();
+
+        bytes.writeInt(offset, value);
+    }
+
+    @Override
+    public int getVolatileValue() {
+        throwExceptionIfClosed();
+
+        return bytes.readVolatileInt(offset);
+    }
+
+    @Override
+    public void setOrderedValue(int value) {
+        throwExceptionIfClosedInSetter();
+
+        bytes.writeOrderedInt(offset, value);
+    }
+
+    @Override
+    public int addValue(int delta) {
+        throwExceptionIfClosed();
+
+        return bytes.addAndGetInt(offset, delta);
+    }
+
+    @Override
+    public int addAtomicValue(int delta) {
+        throwExceptionIfClosed();
+
+        return addValue(delta);
+    }
+
+    @Override
+    public boolean compareAndSwapValue(int expected, int value) {
+        throwExceptionIfClosed();
+
+        return bytes.compareAndSwapInt(offset, expected, value);
+    }
+}

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/ref/BinaryLongArrayReference.java

@@ -0,0 +1,282 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes.ref;
+
+import net.openhft.chronicle.bytes.*;
+import net.openhft.chronicle.core.io.IORuntimeException;
+import net.openhft.chronicle.core.values.LongValue;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+import java.lang.ref.WeakReference;
+import java.nio.BufferOverflowException;
+import java.nio.BufferUnderflowException;
+import java.util.Collections;
+import java.util.IdentityHashMap;
+import java.util.Set;
+
+import static net.openhft.chronicle.bytes.ref.BinaryLongReference.LONG_NOT_COMPLETE;
+
+/**
+ * This class acts a Binary array of 64-bit values. c.f. TextLongArrayReference
+ */
+@SuppressWarnings("rawtypes")
+public class BinaryLongArrayReference extends AbstractReference implements ByteableLongArrayValues, BytesMarshallable {
+    public static final int SHIFT = 3;
+    private static final long CAPACITY = 0;
+    private static final long USED = CAPACITY + Long.BYTES;
+    private static final long VALUES = USED + Long.BYTES;
+    private static final int MAX_TO_STRING = 1024;
+
+    private static Set<WeakReference<BinaryLongArrayReference>> binaryLongArrayReferences = null;
+    private long length;
+
+    public BinaryLongArrayReference() {
+        this(0);
+    }
+
+    public BinaryLongArrayReference(long defaultCapacity) {
+        this.length = (defaultCapacity << SHIFT) + VALUES;
+    }
+
+    public static void startCollecting() {
+        binaryLongArrayReferences = Collections.newSetFromMap(new IdentityHashMap<>());
+    }
+
+    public static void forceAllToNotCompleteState() {
+        binaryLongArrayReferences.forEach(x -> {
+             BinaryLongArrayReference binaryLongReference = x.get();
+            if (binaryLongReference != null) {
+                binaryLongReference.setValueAt(0, LONG_NOT_COMPLETE);
+            }
+        });
+        binaryLongArrayReferences = null;
+    }
+
+    public static void write( Bytes bytes, long capacity) throws BufferOverflowException, IllegalArgumentException {
+        assert (bytes.writePosition() & 0x7) == 0;
+
+        bytes.writeLong(capacity);
+        bytes.writeLong(0L); // used
+        long start = bytes.writePosition();
+        bytes.zeroOut(start, start + (capacity << SHIFT));
+        bytes.writeSkip(capacity << SHIFT);
+    }
+
+    public static void lazyWrite( Bytes bytes, long capacity) throws BufferOverflowException {
+        assert (bytes.writePosition() & 0x7) == 0;
+
+        bytes.writeLong(capacity);
+        bytes.writeLong(0L); // used
+        bytes.writeSkip(capacity << SHIFT);
+    }
+
+    public static long peakLength( BytesStore bytes, long offset) throws BufferUnderflowException {
+        final long capacity = bytes.readLong(offset + CAPACITY);
+        assert capacity > 0 : "capacity too small " + capacity;
+        return (capacity << SHIFT) + VALUES;
+    }
+
+    @Override
+    public long getCapacity() {
+        throwExceptionIfClosed();
+
+        return (length - VALUES) >>> SHIFT;
+    }
+
+    @Override
+    public long getUsed() {
+        throwExceptionIfClosed();
+
+        return bytes.readVolatileLong(offset + USED);
+    }
+
+    @Override
+    public void setMaxUsed(long usedAtLeast) {
+        throwExceptionIfClosedInSetter();
+
+        bytes.writeMaxLong(offset + USED, usedAtLeast);
+    }
+
+    @Override
+    public long getValueAt(long index) throws BufferUnderflowException {
+        throwExceptionIfClosed();
+
+        return bytes.readLong(VALUES + offset + (index << SHIFT));
+    }
+
+    @Override
+    public void setValueAt(long index, long value) throws IllegalArgumentException, BufferOverflowException {
+        throwExceptionIfClosedInSetter();
+
+        bytes.writeLong(VALUES + offset + (index << SHIFT), value);
+    }
+
+    @Override
+    public long getVolatileValueAt(long index) throws BufferUnderflowException {
+        throwExceptionIfClosed();
+
+        return bytes.readVolatileLong(VALUES + offset + (index << SHIFT));
+    }
+
+    @Override
+    public void bindValueAt(long index,  LongValue value) {
+        throwExceptionIfClosed();
+
+        ((BinaryLongReference) value).bytesStore(bytes, VALUES + offset + (index << SHIFT), 8);
+    }
+
+    @Override
+    public void setOrderedValueAt(long index, long value) throws IllegalArgumentException, BufferOverflowException {
+        throwExceptionIfClosedInSetter();
+
+        bytes.writeOrderedLong(VALUES + offset + (index << SHIFT), value);
+    }
+
+    @Override
+    public void bytesStore( BytesStore bytes, long offset, long length) throws BufferUnderflowException, IllegalArgumentException {
+        throwExceptionIfClosed();
+
+        if (length != peakLength(bytes, offset))
+            throw new IllegalArgumentException(length + " != " + peakLength(bytes, offset));
+
+        assert (offset & 7) == 0 : "offset=" + offset;
+        super.bytesStore(bytes, (offset + 7) & ~7, length);
+        this.length = length;
+    }
+
+    @Override
+    public void readMarshallable(BytesIn bytes) throws IORuntimeException {
+        throwExceptionIfClosedInSetter();
+
+        long position = bytes.readPosition();
+        long capacity = bytes.readLong();
+        long used = bytes.readLong();
+        if (capacity < 0 || capacity > bytes.readRemaining() >> SHIFT)
+            throw new IORuntimeException("Corrupt used capacity");
+
+        if (used < 0 || used > capacity)
+            throw new IORuntimeException("Corrupt used value");
+
+        bytes.readSkip(capacity << SHIFT);
+        long length = bytes.readPosition() - position;
+        bytesStore(((Bytes) bytes).bytesStore(), position, length);
+    }
+
+    @Override
+    public void writeMarshallable(BytesOut bytes) {
+        BytesStore bytesStore = bytesStore();
+        if (bytesStore == null) {
+            long capacity = getCapacity();
+            bytes.writeLong(capacity);
+            bytes.writeLong(0);
+            bytes.writeSkip(capacity << SHIFT);
+        } else {
+            bytes.write(bytesStore, offset, length);
+        }
+    }
+
+    @Override
+    public boolean isNull() {
+        throwExceptionIfClosed();
+
+        return bytes == null;
+    }
+
+    @Override
+    public void reset() {
+        throwExceptionIfClosedInSetter();
+
+        bytes = null;
+        offset = 0;
+        length = 0;
+    }
+
+
+    @Override
+    public BytesStore bytesStore() {
+        return bytes;
+    }
+
+    @Override
+    public long offset() {
+        return offset;
+    }
+
+    @Override
+    public long maxSize() {
+        return length;
+    }
+
+
+    public String toString() {
+        if (bytes == null)
+            return "not set";
+         StringBuilder sb = new StringBuilder();
+        sb.append("used: ");
+        long used = getUsed();
+        sb.append(used);
+        sb.append(", value: ");
+         String sep = "";
+        try {
+            int i, max = (int) Math.min(used, Math.min(getCapacity(), MAX_TO_STRING));
+            for (i = 0; i < max; i++) {
+                long valueAt = getValueAt(i);
+                sb.append(sep).append(valueAt);
+                sep = ", ";
+            }
+            if (i < getCapacity())
+                sb.append(" ...");
+
+        } catch (BufferUnderflowException e) {
+            sb.append(" ").append(e);
+        }
+        return sb.toString();
+    }
+
+    @Override
+    public long sizeInBytes(long capacity) {
+        throwExceptionIfClosed();
+
+        return (capacity << SHIFT) + VALUES;
+    }
+
+    @Override
+    public ByteableLongArrayValues capacity(long arrayLength) {
+        throwExceptionIfClosedInSetter();
+
+        BytesStore bytesStore = bytesStore();
+        long length = sizeInBytes(arrayLength);
+        if (bytesStore == null) {
+            this.length = length;
+        } else {
+            assert this.length == length;
+        }
+        return this;
+    }
+
+    @Override
+    public boolean compareAndSet(long index, long expected, long value) throws IllegalArgumentException, BufferOverflowException {
+        throwExceptionIfClosed();
+
+        if (value == LONG_NOT_COMPLETE && binaryLongArrayReferences != null)
+            binaryLongArrayReferences.add(new WeakReference<>(this));
+        return bytes.compareAndSwapLong(VALUES + offset + (index << SHIFT), expected, value);
+    }
+}
+

datastructures-bytes/src/main/java/net/openhft/chronicle/bytes/ref/BinaryLongReference.java

@@ -0,0 +1,136 @@
+/*
+ * Copyright 2016-2020 chronicle.software
+ *
+ * https://chronicle.software
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package net.openhft.chronicle.bytes.ref;
+
+import net.openhft.chronicle.bytes.BytesStore;
+import org.jetbrains.annotations.NotNull;
+
+import java.nio.BufferOverflowException;
+import java.nio.BufferUnderflowException;
+
+public class BinaryLongReference extends AbstractReference implements LongReference {
+    public static final long LONG_NOT_COMPLETE = -1;
+
+    @SuppressWarnings("rawtypes")
+    @Override
+    public void bytesStore( final BytesStore bytes, final long offset, final long length) throws IllegalStateException, IllegalArgumentException, BufferOverflowException, BufferUnderflowException {
+        throwExceptionIfClosed();
+
+        if (length != maxSize())
+            throw new IllegalArgumentException();
+
+        super.bytesStore(bytes, offset, length);
+    }
+
+    @Override
+    public long maxSize() {
+        return 8;
+    }
+
+
+    public String toString() {
+        return bytes == null ? "bytes is null" : "value: " + getValue();
+    }
+
+    @Override
+    public long getValue() {
+        return bytes == null ? 0L : bytes.readLong(offset);
+    }
+
+    @Override
+    public void setValue(long value) {
+        try {
+            bytes.writeLong(offset, value);
+        } catch (NullPointerException e) {
+            throwExceptionIfClosed();
+
+            throw e;
+        }
+    }
+
+    @Override
+    public long getVolatileValue() {
+        try {
+            return bytes.readVolatileLong(offset);
+        } catch (NullPointerException e) {
+            throwExceptionIfClosed();
+
+            throw e;
+        }
+    }
+
+    @Override
+    public void setVolatileValue(long value) {
+        try {
+            bytes.writeVolatileLong(offset, value);
+        } catch (NullPointerException e) {
+            throwExceptionIfClosed();
+
+            throw e;
+        }
+    }
+
+    @Override
+    public long getVolatileValue(long closedValue) {
+        if (isClosed())
+            return closedValue;
+        try {
+            return getVolatileValue();
+        } catch (Exception e) {
+            return closedValue;
+        }
+    }
+
+    @Override
+    public void setOrderedValue(long value) {
+        try {
+            bytes.writeOrderedLong(offset, value);
+        } catch (NullPointerException e) {
+            throwExceptionIfClosed();
+
+            throw e;
+        }
+    }
+
+    @Override
+    public long addValue(long delta) {
+        try {
+            return bytes.addAndGetLong(offset, delta);
+        } catch (NullPointerException e) {
+            throwExceptionIfClosed();
+
+            throw e;
+        }
+    }
+
+    @Override
+    public long addAtomicValue(long delta) {
+        return addValue(delta);
+    }
+
+    @Override
+    public boolean compareAndSwapValue(long expected, long value) {
+        try {
+            return bytes.compareAndSwapLong(offset, expected, value);
+        } catch (NullPointerException e) {
+            throwExceptionIfClosed();
+
+            throw e;
+        }
+    }
+}