Bitcoin Trading Algorithm Back Testing Project Historical Data Setup

I have the desire to create a bitcoin algorithmic trader which will work autonomously without supervision.  However, it might be a wise idea to initially test this trader out with a backtesting solution first.  Once the back testing proves profitable, I can then think about the actual autonomous side of the trading.

That said, I have access to historical trade data from the Coinbase Pro (formerly GDAX) exchange.  The data is not the full order book, but a very simplified view which shows the "Ticker" after each "Match".  For my backtesting application this should be adequate, and the trading strategy that I intend to employ will only need the ticker information.

Goal of this phase:

• Organize the Ticker data I have into a usable Backtesting structure
• Create an API that will allow a Backtesting Application to request the next Tick, or skip an entire day of Ticks if the daily high or daily low is tighter (narrower) than the autonomous trader is quoting.
• Develop unit tests and check the overall performance of the system

Thinking backwards from the end solution, will allow for a usable design.  In the end API, it would be nice to be able have:

   An autonomous Trader;  A "Tick Provider"; and a "Quote Handler" ->  Produce a List of Trades and a Profit and Loss Report

The current task at hand is to develop the "Tick Provider".  So taking a quick look at the data, it comes in "Daily Files" as:

Ticker_20190928.gz Ticker_20191204.gz

Ticker_20190929.gz Ticker_20191205.gz

Ticker_20190930.gz Ticker_20191206.gz

Ticker_20191001.gz Ticker_20191207.gz

Ticker_20191002.gz Ticker_20191208.gz

Ticker_20191003.gz Ticker_20191209.gz

Ticker_20191004.gz Ticker_20191210.gz

Ticker_20191005.gz Ticker_20191211.gz

Each file is a compressed set of "JSON" data with many different "Products" in it. An example of a few rows of json are shown below:

{
  "gdaxv": "tpid",
  "type": "ticker",
  "trade_id": 2577287,
  "sequence": 488009902,
  "time": "2019-09-28T05:21:19.586000Z",
  "product_id": "ETC-USD",
  "price": "4.73",
  "side": "sell",
  "last_size": "2551.79500000",
  "best_bid": "4.728",
  "best_ask": "4.731"}
{
  "gdaxv": "tpid",
  "type": "ticker",
  "trade_id": 75022004,
  "sequence": 10913413268,
  "time": "2019-09-28T05:22:14.457000Z",
  "product_id": "BTC-USD",
  "price": "8165.28",
  "side": "buy",
  "last_size": "0.00585052",
  "best_bid": "8165.27",
  "best_ask": "8165.28"}

The goal for this segment of the overall project will be the following:

1. Identify if a daily summary and extraction exists for the desired product/day.  
1. If Not Create the daily summary
2. Use the existing daily summary
2. Provided a request based feed of the price ticks for the requested product.
3. Allow for an entire day of data to be skipped if the Max or Min price are tighter that required by the client using the data (in this case the autonomous trader may choose to list trade bounds)

The basic API will work a follows:

• Initialize the Exchange via the constructor and then calling the init() method.  I separated out the init method because it could raise an exception and does disk io work like looking for files and making sure that your cache is pre-built.  The idea here is that you will want to run a series of tests against the same product data and same time range.  The first setup might take a bit, but then after that the data will get processed faster.
  
  A sample header will look like this:

{
  "day": "20190929",
  "product": "BCH-USD",
  "open": 227.63,
  "close": 218.17,
  "high": 228.11,
  "low": 212.0}

• Followed by all the trades that match the given product.
• After init() is called, then Exchange.next() can be called repeatedly until a NULL is returned indicating the last trade in the set for analysis.
• This can be used by an autonomous trader to calculate a position or re-quote.  Currently it would be too much work to make this back tester behave exactly as an really exchange API  But this will be good enough for simple tests.

The EXCHANGE CODE:

/* 

* Copyright (c) 2020. Ecocrypt.com 

* 

* Licensed under the Apache License, Version 2.0 (the "License"); 

* you may not use this file except in compliance with the License. 

* You may obtain a copy of the License at 

* 

*     http://www.apache.org/licenses/LICENSE-2.0 

* 

* Unless required by applicable law or agreed to in writing, software 

* distributed under the License is distributed on an "AS IS" BASIS, 

* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 

* See the License for the specific language governing permissions and 

* limitations under the License. 

*/
package com.ecocrypt.backtesting.exchange;

import com.google.gson.Gson;

import java.io.*;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.*;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.logging.Logger;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;

public class ExchangeFeed {

    private static final Logger LOGGER = Logger.getLogger(ExchangeFeed.class.getName());


    private final Date startDate;
    private final Date endDate;
    private final List<String> fileDateStrings = new ArrayList<>();
    private final Calendar calendar;
    private final String cacheDir;
    private final String dataDir;
    private final String product;
    private final AtomicBoolean isReady = new AtomicBoolean(false);

    private Gson gson = Util.gson();
    private Queue<String> cacheQueue = new LinkedList<>();
    private BufferedReader currentFileReader = null;


    public ExchangeFeed(String pStart, String pEnd,String pDataDir, String pCacheDir, String pProduct){

        cacheDir=pCacheDir;
        dataDir=pDataDir;
        product=pProduct;
        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd");
        Date tStartDate = new Date();
        Date tEndDate = new Date();

        try {
            tStartDate = sdf.parse(pStart);
            tEndDate = sdf.parse(pEnd);

        } catch (ParseException e) {

            e.printStackTrace();
        }
        startDate = tStartDate;
        endDate=tEndDate;
        calendar = new GregorianCalendar();
        calendar.setTime(startDate);


        while (calendar.getTimeInMillis()<=endDate.getTime()){
            fileDateStrings.add(Util.yyyyMMdd(calendar));
            calendar.add(Calendar.DATE, 1);
        }
    }

    public void init() throws IOException {
        checkCache();
        cacheQueue.addAll(fileDateStrings);
        progressReaderToNextFile();
        isReady.set(true);
    }

    public Ticker next(){
        String result = null;
        if(isReady.get()){
            if(currentFileReader!=null){
                try {
                    result = currentFileReader.readLine();
                } catch (IOException e) {
                    e.printStackTrace();
                }
                if(result==null){
                    try {
                        progressReaderToNextFile();
                        if(currentFileReader!=null) {
                            result = currentFileReader.readLine();
                        }
                    } catch (IOException e) {
                        e.printStackTrace();
                    }
                }
            }
        }else{
            throw new IllegalStateException(" Call initialize before for first use...");
        }
        Ticker resultTicker = null;
        if(result!=null){
            resultTicker = gson.fromJson(result, Ticker.class);
        }
        return resultTicker;
    }



    void printFileDateStrings(){
        fileDateStrings.stream().forEach(System.out::println);
    }

    void checkCache(){
        fileDateStrings.stream().forEach(s -> {
            File cacheFile = new File(cacheDir+ File.separator+product+"_"+s+".gz");
            if(cacheFile.exists()){
                System.out.println("Using Cache Data "+ cacheFile.getAbsoluteFile());
            }else{
                File rawFile = new File(dataDir+File.separator+"Ticker_"+s+".gz");
                if(rawFile.exists()) {
                    System.out.println("Processing " + rawFile
                            .getAbsoluteFile() +"\n\t INTO Cache -> " + cacheFile.getAbsoluteFile());
                    if(processRawFile(rawFile,cacheFile,s)!=0){
                        // Error processing Data.
                    }
                }else{
                    System.out.println("RAW DATA FILE MISSING " + rawFile.getAbsoluteFile());

                }
            }
        });
    }

    private void progressReaderToNextFile() throws IOException {
        String nextDate = cacheQueue.poll();
        if(nextDate!=null){
            currentFileReader = new BufferedReader(new InputStreamReader(new GZIPInputStream(new FileInputStream(new File(cacheDir+File.separator+product+"_"+nextDate+".gz")))));
            String header = currentFileReader.readLine();
            System.out.println("Header: " + header);
        }else{
            currentFileReader =null;
        }
    }



    private int processRawFile(File rawFile, File cacheFile, String s) {
        int result = -1;

        String tempFile = cacheDir+File.separator+"temp_"+product+"_"+s+".gz";
        OCHLDailyStat stats = new OCHLDailyStat();
        stats.setProduct(product);
        stats.setDay(s);
        Gson g = Util.gson();
        AtomicBoolean firstTrade = new AtomicBoolean(false);
        try (GZIPInputStream inputStream = new GZIPInputStream(new FileInputStream(rawFile));
            BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream));
            GZIPOutputStream os = new GZIPOutputStream(new FileOutputStream(tempFile));
            BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(os))) {


            String line = reader.readLine();

            while(line!=null){
                Ticker t = g.fromJson(line, Ticker.class);

                if(product.equalsIgnoreCase(t.getProduct())){
                    if(firstTrade.compareAndSet(false, true)){
                        stats.setOpen(t.getPrice());
                        stats.setHigh(t.getPrice());
                        stats.setLow(t.getPrice());
                    }
                    bw.write(line);
                    bw.newLine();
                    if(t.getPrice()<stats.getLow()){
                        stats.setLow(t.getPrice());
                    }
                    if(t.getPrice()>stats.getHigh()){
                        stats.setHigh(t.getPrice());
                    }
                    stats.setClose(t.getPrice());
                }
                line = reader.readLine();
            }

            bw.flush();
        } catch (IOException e) {
            e.printStackTrace();
        }
        try(BufferedOutputStream bos = new BufferedOutputStream(new GZIPOutputStream(new FileOutputStream(cacheFile)));
        BufferedInputStream bis = new BufferedInputStream(new GZIPInputStream(new FileInputStream(tempFile)))){

            String statsAsString = g.toJson(stats);
            statsAsString +="\n";
            bos.write(statsAsString.getBytes());
            byte[] buf = new byte[1024];
            int b = -1;

            while((b = bis.read(buf)) >-1){
                bos.write(buf,0,b);
            }
            bos.flush();
            result =0;

        }catch (IOException e){
            e.printStackTrace();
        }
        File f = new File(tempFile);
        f.delete();

        return result;
    }
}

One of the key tricks here was using a TypeAdapter for the Gson Json Parser the type that needed to be serialized and de-serialized was an ENUM called SIDE.  This has 2 valid options either BUY or SELL.  

/* 

* Copyright (c) 2020. Ecocrypt.com 

* 

* Licensed under the Apache License, Version 2.0 (the "License"); 

* you may not use this file except in compliance with the License. 

* You may obtain a copy of the License at 

* 

*     http://www.apache.org/licenses/LICENSE-2.0 

* 

* Unless required by applicable law or agreed to in writing, software 

* distributed under the License is distributed on an "AS IS" BASIS, 

* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 

* See the License for the specific language governing permissions and 

* limitations under the License. 

*/

package com.ecocrypt.backtesting.exchange;

import com.google.gson.*;

import java.lang.reflect.Type;

public class SIDETypeAdapter implements JsonSerializer<SIDE>,
        JsonDeserializer<SIDE> {


        public SIDETypeAdapter() {
        }


    @Override    public SIDE deserialize(JsonElement jsonElement, Type type, JsonDeserializationContext jsonDeserializationContext) throws JsonParseException {
        return SIDE.get(jsonElement.getAsString());
    }

    @Override    public JsonElement serialize(SIDE side, Type type, JsonSerializationContext jsonSerializationContext) {
        return new JsonPrimitive(side.toString());
    }
}

Then using the Gson TypeAdapter is done in the GsonBuilder as follows:

package com.ecocrypt.backtesting.exchange;

import com.google.gson.Gson;
import com.google.gson.GsonBuilder;

import java.text.SimpleDateFormat;
import java.util.Calendar;
import java.util.Date;

public class Util {

    public static Gson gson(){
        Gson g = new GsonBuilder()
                .setDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSZ")
                .registerTypeAdapter(SIDE.class, new SIDETypeAdapter())
                .create();
        return g;
    }

    public static String yyyyMMdd(Calendar c){
        return yyyyMMdd(c.getTime());
    }

    private static String yyyyMMdd(Date time) {

        SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMdd");
        return sdf.format(time);

    }
    private static String yyyyMMdd(long l){
        return yyyyMMdd(new Date(l));
    }
}

The SIDE enum is show for completeness:

package com.ecocrypt.backtesting.exchange;

public enum SIDE {
    BUY {
        @Override        public String toString() {
            return "BUY";
        }
    },
    SELL{
        @Override        public String toString() {
            return "SELL";
        }
    };
    public static SIDE get(String s){
        if(s.equalsIgnoreCase("BUY")){
            return BUY;
        }else{
            return SELL;
        }
    }
}