King's View of the Battle

 Open GL model of a chess board.

Bitcoin Trading Algorithm Back Testing Project Historical Data Setup

I have the desire to create a bitcoin algorithmic trader which will work autonomously without supervision.  However, it might be a wise idea to initially test this trader out with a backtesting solution first.  Once the back testing proves profitable, I can then think about the actual autonomous side of the trading.

That said, I have access to historical trade data from the Coinbase Pro (formerly GDAX) exchange.  The data is not the full order book, but a very simplified view which shows the "Ticker" after each "Match".  For my backtesting application this should be adequate, and the trading strategy that I intend to employ will only need the ticker information.

Goal of this phase:

• Organize the Ticker data I have into a usable Backtesting structure
• Create an API that will allow a Backtesting Application to request the next Tick, or skip an entire day of Ticks if the daily high or daily low is tighter (narrower) than the autonomous trader is quoting.
• Develop unit tests and check the overall performance of the system

Thinking backwards from the end solution, will allow for a usable design.  In the end API, it would be nice to be able have:

   An autonomous Trader;  A "Tick Provider"; and a "Quote Handler" ->  Produce a List of Trades and a Profit and Loss Report

The current task at hand is to develop the "Tick Provider".  So taking a quick look at the data, it comes in "Daily Files" as:

Ticker_20190928.gz Ticker_20191204.gz

Ticker_20190929.gz Ticker_20191205.gz

Ticker_20190930.gz Ticker_20191206.gz

Ticker_20191001.gz Ticker_20191207.gz

Ticker_20191002.gz Ticker_20191208.gz

Ticker_20191003.gz Ticker_20191209.gz

Ticker_20191004.gz Ticker_20191210.gz

Ticker_20191005.gz Ticker_20191211.gz

Each file is a compressed set of "JSON" data with many different "Products" in it. An example of a few rows of json are shown below:

{
  "gdaxv": "tpid",
  "type": "ticker",
  "trade_id": 2577287,
  "sequence": 488009902,
  "time": "2019-09-28T05:21:19.586000Z",
  "product_id": "ETC-USD",
  "price": "4.73",
  "side": "sell",
  "last_size": "2551.79500000",
  "best_bid": "4.728",
  "best_ask": "4.731"}
{
  "gdaxv": "tpid",
  "type": "ticker",
  "trade_id": 75022004,
  "sequence": 10913413268,
  "time": "2019-09-28T05:22:14.457000Z",
  "product_id": "BTC-USD",
  "price": "8165.28",
  "side": "buy",
  "last_size": "0.00585052",
  "best_bid": "8165.27",
  "best_ask": "8165.28"}

The goal for this segment of the overall project will be the following:

1. Identify if a daily summary and extraction exists for the desired product/day.  
1. If Not Create the daily summary
2. Use the existing daily summary
2. Provided a request based feed of the price ticks for the requested product.
3. Allow for an entire day of data to be skipped if the Max or Min price are tighter that required by the client using the data (in this case the autonomous trader may choose to list trade bounds)

The basic API will work a follows:

• Initialize the Exchange via the constructor and then calling the init() method.  I separated out the init method because it could raise an exception and does disk io work like looking for files and making sure that your cache is pre-built.  The idea here is that you will want to run a series of tests against the same product data and same time range.  The first setup might take a bit, but then after that the data will get processed faster.
  
  A sample header will look like this:

{
  "day": "20190929",
  "product": "BCH-USD",
  "open": 227.63,
  "close": 218.17,
  "high": 228.11,
  "low": 212.0}

• Followed by all the trades that match the given product.
• After init() is called, then Exchange.next() can be called repeatedly until a NULL is returned indicating the last trade in the set for analysis.
• This can be used by an autonomous trader to calculate a position or re-quote.  Currently it would be too much work to make this back tester behave exactly as an really exchange API  But this will be good enough for simple tests.

The EXCHANGE CODE:

/* 

* Copyright (c) 2020. Ecocrypt.com 

* 

* Licensed under the Apache License, Version 2.0 (the "License"); 

* you may not use this file except in compliance with the License. 

* You may obtain a copy of the License at 

* 

*     http://www.apache.org/licenses/LICENSE-2.0 

* 

* Unless required by applicable law or agreed to in writing, software 

* distributed under the License is distributed on an "AS IS" BASIS, 

* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 

* See the License for the specific language governing permissions and 

* limitations under the License. 

*/
package com.ecocrypt.backtesting.exchange;

import com.google.gson.Gson;

import java.io.*;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.*;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.logging.Logger;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;

public class ExchangeFeed {

    private static final Logger LOGGER = Logger.getLogger(ExchangeFeed.class.getName());


    private final Date startDate;
    private final Date endDate;
    private final List<String> fileDateStrings = new ArrayList<>();
    private final Calendar calendar;
    private final String cacheDir;
    private final String dataDir;
    private final String product;
    private final AtomicBoolean isReady = new AtomicBoolean(false);

    private Gson gson = Util.gson();
    private Queue<String> cacheQueue = new LinkedList<>();
    private BufferedReader currentFileReader = null;


    public ExchangeFeed(String pStart, String pEnd,String pDataDir, String pCacheDir, String pProduct){

        cacheDir=pCacheDir;
        dataDir=pDataDir;
        product=pProduct;
        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd");
        Date tStartDate = new Date();
        Date tEndDate = new Date();

        try {
            tStartDate = sdf.parse(pStart);
            tEndDate = sdf.parse(pEnd);

        } catch (ParseException e) {

            e.printStackTrace();
        }
        startDate = tStartDate;
        endDate=tEndDate;
        calendar = new GregorianCalendar();
        calendar.setTime(startDate);


        while (calendar.getTimeInMillis()<=endDate.getTime()){
            fileDateStrings.add(Util.yyyyMMdd(calendar));
            calendar.add(Calendar.DATE, 1);
        }
    }

    public void init() throws IOException {
        checkCache();
        cacheQueue.addAll(fileDateStrings);
        progressReaderToNextFile();
        isReady.set(true);
    }

    public Ticker next(){
        String result = null;
        if(isReady.get()){
            if(currentFileReader!=null){
                try {
                    result = currentFileReader.readLine();
                } catch (IOException e) {
                    e.printStackTrace();
                }
                if(result==null){
                    try {
                        progressReaderToNextFile();
                        if(currentFileReader!=null) {
                            result = currentFileReader.readLine();
                        }
                    } catch (IOException e) {
                        e.printStackTrace();
                    }
                }
            }
        }else{
            throw new IllegalStateException(" Call initialize before for first use...");
        }
        Ticker resultTicker = null;
        if(result!=null){
            resultTicker = gson.fromJson(result, Ticker.class);
        }
        return resultTicker;
    }



    void printFileDateStrings(){
        fileDateStrings.stream().forEach(System.out::println);
    }

    void checkCache(){
        fileDateStrings.stream().forEach(s -> {
            File cacheFile = new File(cacheDir+ File.separator+product+"_"+s+".gz");
            if(cacheFile.exists()){
                System.out.println("Using Cache Data "+ cacheFile.getAbsoluteFile());
            }else{
                File rawFile = new File(dataDir+File.separator+"Ticker_"+s+".gz");
                if(rawFile.exists()) {
                    System.out.println("Processing " + rawFile
                            .getAbsoluteFile() +"\n\t INTO Cache -> " + cacheFile.getAbsoluteFile());
                    if(processRawFile(rawFile,cacheFile,s)!=0){
                        // Error processing Data.
                    }
                }else{
                    System.out.println("RAW DATA FILE MISSING " + rawFile.getAbsoluteFile());

                }
            }
        });
    }

    private void progressReaderToNextFile() throws IOException {
        String nextDate = cacheQueue.poll();
        if(nextDate!=null){
            currentFileReader = new BufferedReader(new InputStreamReader(new GZIPInputStream(new FileInputStream(new File(cacheDir+File.separator+product+"_"+nextDate+".gz")))));
            String header = currentFileReader.readLine();
            System.out.println("Header: " + header);
        }else{
            currentFileReader =null;
        }
    }



    private int processRawFile(File rawFile, File cacheFile, String s) {
        int result = -1;

        String tempFile = cacheDir+File.separator+"temp_"+product+"_"+s+".gz";
        OCHLDailyStat stats = new OCHLDailyStat();
        stats.setProduct(product);
        stats.setDay(s);
        Gson g = Util.gson();
        AtomicBoolean firstTrade = new AtomicBoolean(false);
        try (GZIPInputStream inputStream = new GZIPInputStream(new FileInputStream(rawFile));
            BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream));
            GZIPOutputStream os = new GZIPOutputStream(new FileOutputStream(tempFile));
            BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(os))) {


            String line = reader.readLine();

            while(line!=null){
                Ticker t = g.fromJson(line, Ticker.class);

                if(product.equalsIgnoreCase(t.getProduct())){
                    if(firstTrade.compareAndSet(false, true)){
                        stats.setOpen(t.getPrice());
                        stats.setHigh(t.getPrice());
                        stats.setLow(t.getPrice());
                    }
                    bw.write(line);
                    bw.newLine();
                    if(t.getPrice()<stats.getLow()){
                        stats.setLow(t.getPrice());
                    }
                    if(t.getPrice()>stats.getHigh()){
                        stats.setHigh(t.getPrice());
                    }
                    stats.setClose(t.getPrice());
                }
                line = reader.readLine();
            }

            bw.flush();
        } catch (IOException e) {
            e.printStackTrace();
        }
        try(BufferedOutputStream bos = new BufferedOutputStream(new GZIPOutputStream(new FileOutputStream(cacheFile)));
        BufferedInputStream bis = new BufferedInputStream(new GZIPInputStream(new FileInputStream(tempFile)))){

            String statsAsString = g.toJson(stats);
            statsAsString +="\n";
            bos.write(statsAsString.getBytes());
            byte[] buf = new byte[1024];
            int b = -1;

            while((b = bis.read(buf)) >-1){
                bos.write(buf,0,b);
            }
            bos.flush();
            result =0;

        }catch (IOException e){
            e.printStackTrace();
        }
        File f = new File(tempFile);
        f.delete();

        return result;
    }
}

One of the key tricks here was using a TypeAdapter for the Gson Json Parser the type that needed to be serialized and de-serialized was an ENUM called SIDE.  This has 2 valid options either BUY or SELL.  

/* 

* Copyright (c) 2020. Ecocrypt.com 

* 

* Licensed under the Apache License, Version 2.0 (the "License"); 

* you may not use this file except in compliance with the License. 

* You may obtain a copy of the License at 

* 

*     http://www.apache.org/licenses/LICENSE-2.0 

* 

* Unless required by applicable law or agreed to in writing, software 

* distributed under the License is distributed on an "AS IS" BASIS, 

* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 

* See the License for the specific language governing permissions and 

* limitations under the License. 

*/

package com.ecocrypt.backtesting.exchange;

import com.google.gson.*;

import java.lang.reflect.Type;

public class SIDETypeAdapter implements JsonSerializer<SIDE>,
        JsonDeserializer<SIDE> {


        public SIDETypeAdapter() {
        }


    @Override    public SIDE deserialize(JsonElement jsonElement, Type type, JsonDeserializationContext jsonDeserializationContext) throws JsonParseException {
        return SIDE.get(jsonElement.getAsString());
    }

    @Override    public JsonElement serialize(SIDE side, Type type, JsonSerializationContext jsonSerializationContext) {
        return new JsonPrimitive(side.toString());
    }
}

Then using the Gson TypeAdapter is done in the GsonBuilder as follows:

package com.ecocrypt.backtesting.exchange;

import com.google.gson.Gson;
import com.google.gson.GsonBuilder;

import java.text.SimpleDateFormat;
import java.util.Calendar;
import java.util.Date;

public class Util {

    public static Gson gson(){
        Gson g = new GsonBuilder()
                .setDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSZ")
                .registerTypeAdapter(SIDE.class, new SIDETypeAdapter())
                .create();
        return g;
    }

    public static String yyyyMMdd(Calendar c){
        return yyyyMMdd(c.getTime());
    }

    private static String yyyyMMdd(Date time) {

        SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMdd");
        return sdf.format(time);

    }
    private static String yyyyMMdd(long l){
        return yyyyMMdd(new Date(l));
    }
}

The SIDE enum is show for completeness:

package com.ecocrypt.backtesting.exchange;

public enum SIDE {
    BUY {
        @Override        public String toString() {
            return "BUY";
        }
    },
    SELL{
        @Override        public String toString() {
            return "SELL";
        }
    };
    public static SIDE get(String s){
        if(s.equalsIgnoreCase("BUY")){
            return BUY;
        }else{
            return SELL;
        }
    }
}

Classic Visitor Pattern in Java

First let's take a quick look at the classic visitor pattern.
When do you want to use the visitor:

• A fixed number of objects that you want to be able to apply operations on.
• These objects don't currently have the ability to perform the operations needed.
• Limited ability access or change the behavior of the objects.

The basic idea of the visitor pattern is this: Starting with a group of related objects say, Shapes, where these shapes may be squares and circles etc.. which do not know how to calculate their perimeters and you either don't have access to or don't want to code the logic in the shape.  Therefore we need a mechanism to calculate the perimeter (or area or other feature) without modifying the original code.  However, we do have the good fortune of having each Shape implement an "accept" method which accepts a visitor.

So we develop a concrete "PerimeterShapeVisitor" which knows how to calculate the perimeter for all known shapes.  This class does not know how to do anything else.  Also this class must implement the special "ShapeVisitor" interface.  This allows polymorphism of the java language to select which shape logic needs to be invoked.  Key code samples are discussed below.  The running code can be found at GIST Classic Visitor
------------------------------------------------------------------------------------------------
Classic Visitor Code

• Each shape must implement the Shape interface to "accept" a visitor e.g.
  

  public interface Shape {
      void accept(ShapeVisitor v);
  }
  
  

• Each ShapeVisitor (e.g. the PerimeterVisitor) must implement the ShapeVisitor
  

  public interface ShapeVisitor {
      void visit(Square s);
      void visit(Circle c);
      void visit(IsoscelesTriangle t);
  }
  
  

• And the Concrete Perimeter Visitor Looks like this:
  

  public class PerimeterVisitor implements ShapeVisitor{
  
      @Override    public void visit(Square s) {
          double p = s.side()*4;
          System.out.println("Square perimeter: " + p);
      }
  
      @Override    public void visit(Circle c) {
          double radius = c.radius();
          double p = Math.PI*2*radius;
          System.out.println("Circle perimeter: " + p);
  
      }
  
      @Override    public void visit(IsoscelesTriangle t) {
          double base = t.base();
          double height = t.height();
          double p = base+Math.sqrt(base*base + 4*height*height);
          System.out.println("Isosolese Triangle perimeter: " + p);
      }
  }

Running the Visitor Test:

public static void main(String[] args){
    List<Shape> shapeList;
    shapeList = new ArrayList<>();

    shapeList.add(new Circle(1));
    shapeList.add(new Square(1));
    shapeList.add(new IsoscelesTriangle(1, 1));
    PerimeterVisitor v = new PerimeterVisitor();
    for (Shape s:shapeList) {
        s.accept(v);
    }

}

Produces the following output:
Circle perimeter: 6.283185307179586
Square perimeter: 4.0
Isosceles Triangle perimeter: 3.23606797749979

SEQUENCE OF EVENTS IN THE VISITOR:

1.  A Shape invokes the accept(v) method passing a visitor associated with whatever operation it would like to perform.
2. Since Shape is an interface, polymorphism of the language will dynamically bind the "accept(v)" method associated with the concrete shape, e.g. for a Circle, the Circle's accept method will be invoked.  This decision is made at run time
3. The accept method will the invoke the "visit" method of the PerimeterShapeVisitor Class, and the method selected will be the Calculate

Performance of the parallelSort method introduced in Java SE 8.

Taking a quick look at a feature added to java in SE8.  This is a parallel Array sort, and it takes advantage of multiple processors to sort arrays more quickly.  The feature is really easy to use, simply invoke Arrays.parallelSort(input);. instead of the old Arrays.sort(input);.  That is it.  Lets look at a simple performance test with results:

            size %Faster               serial ns      parallel ns
              10 -83.5% Faster ->          1,908            3,502
             100 17.0%  Faster ->          5,496            4,560
            1000 65.3%  Faster ->        149,033           51,768
           10000 5.1%   Faster ->        488,044          463,089
          100000 57.7%  Faster ->      6,153,690        2,601,578
         1000000 56.2%  Faster ->     72,704,992       31,827,891
        10000000 58.2%  Faster ->    855,719,416      357,283,539

Some Notes about this test:

• The sort and parallel sort are tested on the exact same Array.  An array is generated using the syntactic sugar of $\lambda$ expressions and then duplicated with a System.arrayCopy(...).  as 

private double[] randomArray(int size) {

    double[] result = new double[size];
    Random randGen = new Random();

    // Use the parallel Set All function to generate this random number quickly.    

    Arrays.parallelSetAll(result, value -> {
        return randGen.nextDouble() * 10000;
    });
    return result;
}

• The test is run 9 times without recording the result.  This is to make sure that the JVM as "warmed up" and the hotspot will not experience a compile pause.
• Prior to each sort a System.gc() is invoked to give the JVM a chance to clean up any garbage.

Looking at the results on a 4 core iMac:

Hardware Overview:

  Model Name: iMac

  Processor Name: Quad-Core Intel Core i5

  Processor Speed: 3.2 GHz

  Number of Processors: 1

  Total Number of Cores: 4

  L2 Cache (per Core): 256 KB

  L3 Cache: 6 MB

  Memory: 32 GB

Based on the above results it is clear that an array must be at least 100 elements in size to overcome the cost of using multiple threads for the sorting, however at 100,000 a consistent cost savings of 50%+ exists.
The gist for this is:

Installing Multiple JVMs on a Mac Manually and Switching Manually with latest Default

This blog will show you now to install multiple version of java manually on a Mac, and switch between them manually.  It will also show how to set and maintain the default java to the "latest" version.

First, Check if there are any version of java installed.  I just bought a 2013 iMac on Ebay, and set it up tabula rosa.

iMac:~ $ /usr/libexec/java_home -V

Unable to find any JVMs matching version "(null)".

Matching Java Virtual Machines (0):

Default Java Virtual Machines (0):

No Java runtime present, try --request to install.

Mac OS looks for java in the /Library/Java/JavaVirtualMachines/ directory.  Therefore if we download a valid Java JDK, and "copy" the un-archived structure over to that directory, we will see that it works.  This is done as follows.

Java 13 is available from Java 13 link - > https://jdk.java.net/13/ or it can obtained via wget with:

iMac:java $ wget https://download.java.net/java/GA/jdk13.0.2/d4173c853231432d94f001e99d882ca7/8/GPL/openjdk-13.0.2_osx-x64_bin.tar.gz

--2020-02-04 19:01:16--  https://download.java.net/java/GA/jdk13.0.2/d4173c853231432d94f001e99d882ca7/8/GPL/openjdk-13.0.2_osx-x64_bin.tar.gz

Resolving download.java.net (download.java.net)... 104.87.12.11

Connecting to download.java.net (download.java.net)|104.87.12.11|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 189969691 (181M) [application/x-gzip]

Saving to: ‘openjdk-13.0.2_osx-x64_bin.tar.gz’

openjdk-13.0.2_osx-x64_bin.tar.gz        7%[====>    ]  13.50M   254KB/s    eta 9m 8s  

Once this has been downloaded:  it can be extracted and copied or move to the /Library/Java/JavaVirtualMachines/ directory.

Extract with the "tar -xvzf openjdk-13.0.2_osx-x64_bin.tar.gz" which says eXtract Verbose Compressed(z) Filesystem.

$ tar -xvzf openjdk-13.0.2_osx-x64_bin.tar.gz
$ mv jdk-13.0.1.jdk/ /Library/Java/JavaVirtualMachines/

Re-running the /usr/libexec/java_home -V command gives:

$ /usr/libexec/java_home -V

Matching Java Virtual Machines (1):

    13.0.1, x86_64: "OpenJDK 13.0.1" /Library/Java/JavaVirtualMachines/jdk-13.0.1.jdk/Contents/Home

/Library/Java/JavaVirtualMachines/jdk-13.0.1.jdk/Contents/Home

And 'java -version' on the command line gives:

$ java -version

openjdk version "13.0.1" 2019-10-15

OpenJDK Runtime Environment (build 13.0.1+9)

OpenJDK 64-Bit Server VM (build 13.0.1+9, mixed mode, sharing)

After downloading and installing several different version of java (which included creating an oracle account, and getting one of the "legacy/archived" version of java from the oracle website) my lib execute for java home looked like this:

$ /usr/libexec/java_home -V

Matching Java Virtual Machines (4):

    13.0.1, x86_64: "OpenJDK 13.0.1" /Library/Java/JavaVirtualMachines/jdk-13.0.1.jdk/Contents/Home

    11.0.6, x86_64: "Java SE 11.0.6" /Library/Java/JavaVirtualMachines/jdk-11.0.6.jdk/Contents/Home

    1.8.0_241, x86_64: "Java SE 8" /Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home

    1.7.0_80, x86_64: "Java SE 7" /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home

/Library/Java/JavaVirtualMachines/jdk-13.0.1.jdk/Contents/Home

Now it might make sense to just add some convenience methods to the environment files.  Basically this is the step to make it very easy to switch to different version of java on the command line:

PLACE THIS IN YOUR ~/.bash_profile or other startup environment script directory

export  JAVA_HOME_7=$(/usr/libexec/java_home -v1.7)

export  JAVA_HOME_8=$(/usr/libexec/java_home -v1.8)

#export JAVA_HOME_9=$(/usr/libexec/java_home -v9)

#export JAVA_HOME_10=$(/usr/libexec/java_home -v10)

export  JAVA_HOME_11=$(/usr/libexec/java_home -v11)

export  JAVA_HOME_13=$(/usr/libexec/java_home -v13)

alias java7='export JAVA_HOME=$JAVA_HOME_7'

alias java8='export JAVA_HOME=$JAVA_HOME_8'

#alias java9='export JAVA_HOME=$JAVA_HOME_9'

#alias java10='export JAVA_HOME=$JAVA_HOME_10'

alias java11='export JAVA_HOME=$JAVA_HOME_11'

#alias java12='export JAVA_HOME=$JAVA_HOME_12'

alias java13='export JAVA_HOME=$JAVA_HOME_13'

#default

java13

Source your newly modified .bash_profile (this only needs to be done once, and not after a login logout...)
$ source ~/.bash_profile
Now switching from java13 to java7 is as easy as typing $ java7 at the command line E.G.

iMac:~ $ java7

iMac:~ $ java -version

java version "1.7.0_80"

Java(TM) SE Runtime Environment (build 1.7.0_80-b15)

Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)

iMac:~ $ java8

iMac:~ $ java -version

java version "1.8.0_241"

Java(TM) SE Runtime Environment (build 1.8.0_241-b07)

Java HotSpot(TM) 64-Bit Server VM (build 25.241-b07, mixed mode)

iMac:~ $ java11

iMac:~ $ java -version

java version "11.0.6" 2020-01-14 LTS

Java(TM) SE Runtime Environment 18.9 (build 11.0.6+8-LTS)

Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.6+8-LTS, mixed mode)

iMac:~ $ java13

iMac:~ $ java -version

openjdk version "13.0.1" 2019-10-15

OpenJDK Runtime Environment (build 13.0.1+9)

OpenJDK 64-Bit Server VM (build 13.0.1+9, mixed mode, sharing)

Fly Swatter Problem Google Code Jam 2008 Qualifying Round Problem 3

The Fly Swatter problem was quite challenging.
This solution uses an "Exact" closed form solution.

Basic problem summary: calculate the probability of hitting a fly with a tennis racket.
CodeJam's Detailed statement:  Code Jam 2008 Qualifying Round P3

The basic idea is the following:

• Divided the Tennis racket into 4 symmetric quadrants
• Calculate the area where the fly would be able to safely fly through
• Calculate the total Area
• Solution 1-safeArea/totalArea is the solution.  format and maintain 6 sig figs.

Prior to really diving into this problem there is a bit of Trigonometry that must be programmed. Specifically, we will be required to calculate the area of a "chord" of a circle.  For the most part equations of this assume that $Area = f(\theta,r) \stackrel{\bullet}{\equiv} \theta  $.  However, we will need this in the form $Area = f(x_{1},x_{2},r) \space where \space x_{1} ,  x_{2}  $ are the x-coordinates of the 2 points of the chord.  So the Area of the chord in a usable format for this problem becomes: $$ { 1 \over {2}} r^2 ( 2 \arcsin{{l}\over{2r}}) - \sin{(2 \arcsin{l\over{2r}})}$$ where $ l $ is the length of the chord.
Or In Code Calculate the Area of a Chord:

The North-East Case:

$$Area = gap^{2}  -  {1 \over 2} (xTwo -  X_{RIM \space yTwo} ) * (yTwo - X_{RIM \space xTwo}) + ChordArea (X_{RIM \space yTwo},X_{RIM \space xTwo})$$,

West East Case:
$$ Area = gap*(yOne - RIM_{y \space xTwo}) + {{1 \over 2} (RIM_{y \space xOne} - RIM_{y \space xTwo})} + ChordArea (X_{RIM \space xTwo},X_{RIM \space xOne})$$
OR IN CODE In Code Area Calc for West-East Case:

For The North South And West South Cases: the code with picture is self explanatory, see the code at the bottom of the blog or on Git HUB

Mountain Lion Dock Freeze.

This is quite frustrating, the Mountain Lion Dock Freezes. WAY to often. What do you do when this happens?

SOME OPTIONS:

1.  HARD REBOOT ( not fun...)
2. killall Dock (reqires access to a terminal)

Because this happens often, a habit has formed to make sure to keep a terminal is always open.  Then just switch via the apple in the upper left, and invoke

PROMPT $ killall Dock

 
Shortest blog post I ever wrote... Just trying to get the word out...

Unique Static Content and Eclipse Non-Consume Regex Glitch

I discovered a technique to allow infinite cache expire time for images and static content on websites. This technique is highly desirable because image requests and 304 requests consume time and make the user experience poor.  The technique is to include a version as follows:

 



-
 

In Fact, in the above example you could be even more sophisticated and include the MD5-checksum or other very unique identifier.
So I cracked open Eclipse to make the REGEX search and replace.  The expression I came up with was:

#Search RegEx
# This looks for .jpg, and then does a 
# NON-Consuming match on a word boundary where there is NOT a ?
# For example .jpg" and .jpg' would match, but .jpg?version2 will not
\.jpg(?\b)(?[^\?])
#Replacement
\.jpg?STATIC_CONTENT_VERSION

I proceeded to do the simple search and replace.

To my chagrin, I received the infamous Error:
, "Match string has changed in the file" 


Why is this?  Well I googled it.  It is a know bug with Eclipse and regexes...
Eclipse Bug.

Workaround:  well we could use Netbeans, but I choose to go more primitive and use perl.  This post will include the generic perl script developed to solve the simple issue and how it was used:

To use, simple type perl findreplace.pl regexSearch regexReplace [0|1]
The 0 is a dryrun, the 1 will really do the replace.

use File::Find;
use File::Basename;

# The inputs:
# directory to start in, findRegex, replacement, dryrun indicator
@DIRS = $ARGV[0];
$patternFind = $ARGV[1];
$patternReplace = $ARGV[2];
$dryrun = $ARGV[3];

#  process a single file (this sub is called from below)
sub processFile{

    $baseN="";
    $dirN="";
    $extN="";
    $linesProcessed = 0;
    $currentFile = $File::Find::name;
    ($baseN,$dirN,$extN) = fileparse($currentFile,'\..*');
    if ($extN eq ".java" || 
        $extN eq ".jsp"  ||
        $extN eq ".properties" ||
        $extN eq ".js" ||
        $extN eq ".css" ){
        # This represents a file that we want to do a search and replace on.
        # print "found java $currentFile :$extN:$baseN:$dirN:\n";
    }else{
        # print "\nRETURNING: $currentFile\n\n";
        return;
    }

    $currentBaseName = $_;

    # OPEN THE FILE READ ONLY.
    open FILE, $currentBaseName;
    $modify= 0;
    # DETERMINE IF THE FILE CONTAINS A MATCH
    while($line =){
        if($line =~ /$patternFind/){
            $modify++
        }
    }
    close FILE;
    if($modify==0){
        # THERE WAS NO REGEX MATCH, SO EXIT.
        return;
    }

    # NO IT's not the evil number, 
    # it means set to global read/write in unix.
    # required for TFS integration.
    chmod 0666, $currentBaseName;

    open (FILE, "+< $currentBaseName")  or die "can't read $currentFile $!";
    $out = '';
    while(){
        if($_ =~ /$patternFind/){
            $linesProcessed++;
        print "\tReplace: $_\n";
            s/$patternFind/$patternReplace/eg;
        print "\tWith: $_\n";
        }

        $out .=$_;
    }

    if($dryrun==1){

        seek(FILE,0,0) or die "can't seek to start of $currentFile $!";
        print FILE $out or die "cant't print $currentFile $!";
        truncate(FILE, tell(FILE))  or die "can't truncate $currentFile: $!";
        close (FILE)         or die "can't close $currentFile: $!";

    }

    print "File:$baseN$extN :From ->:$dirN\n";
    print "Lines Changed: $linesProcessed\n";
    print "___________________________________________________\n";
}
find (\&processFile, @DIRS);

hope you find this useful!

Eclipse/NetBeans Multi Line Search and Replace Across Files

We all know System.out.println statements are not good for performance and clutter logs.  We should use a logger instead.  Loggers allow for the control of logging levels.  How do you replace these System.outs in a large project?  Here are the steps used recently:

1. Come up with a good REGular EXspresion (REGEX), for matching your desired code.
2. Come up with your replacement regex and capture user.
3. Ensure that your code compiles and inspect your changes.

In this case we want to replace the following:

package com.ecokrypt...

public class ToManySystemOuts {

    System.out.println("Single Line System.out.println");


    System.out.println("Rare multiline " +
           " System.out.println ");

}

We need to find a REGEX that will match both of these cases.  There are several ways to do any REGEX, but a solution that is easy  to understand that will work is:

   (?s)System.out.println(\(([^;]+?)\));

We have to dissect what this is saying to understand a couple of subtleties:

• (?s) - indicates that we search past line feeds.  Without this we will not match the second case because the line feed would cause the REGEX to not match at all.
• System.out.println - matches the exact phrase
• (\(([^;]+?)\)) - is very complicated, and will take time before you can read this.  A technique that typically works is to read it inside out.  Trying that out:
• [^;]+ - look for anything that is NOT a ';' (semicolon).  the + is saying that you have to match at least 1, and there is no limit to how many characters you can match.
• ([^;]+?) - Has a real subtlety to it, Regexes have an ability to be greedy, or lazy.  A greedy regex will match the largest region it can.  A lazy regex will match the smallest area it can.  It will take time to figure out when to use which, but the basic advice given here is ALWAYS try to use lazy.  Only use greedy if you really need it.  This REGEX is obviously lazy, how can you tell?  Think about it a bit.
• the ? indicates that you are going to look for the next sequence imediately after.  
• the (...) indicates you are going to "capture" whatever you find presumably for later use.
• therefore the ( ) are not matched but are just part of the regex capture syntax.
• So the answer is: if the ? is inside the capture group e.g. (...?) the matching is lazy; if the ? is outside of the capture group, the matching is greedy e.g. (...)?
• \(([^;]+?)\)  - says, adds the \(...\) to the previous the \ is an escape character and says match a ( and a ), don't create another capture group.
• (\(...\)) - says, now add another capture group containing everything inside the literals ( ).
• ; the semicolon at the end says match a literal semicolon.

That is pretty much all there is to it.  Look at the following REGEX, which might be a common mistake and see if you can figure out why it won't work.  Guess what it will match.  (HINT thing about greedy vs lazy).

   (?s)System.out.println(\(([^;]+)?\));

Now that we have our matching REGEX, we need to get our replacement REGEX:

   System.out.println$1;

This is much simpler, it says:  replace the above match, with logger.debug$1; where $1 is inclusive the literal (...).  e.g. the braces and everything inside of them.

How do we do this inside of Eclipse or NetBeans?
I find NetBeans to be much nicer in some respects, and not so nice in others.

NetBeans in File Search Replace

This is the netbeans view, it uses the space very nicely and there is NO POPUPS...  You really have to like the ergonomics of NetBeans.  The search is dynamic and nicely highlighted so you can see the search as you go.

NetBeans Replace Across All Files
USE CRTL-H for a shortcut to this Menu.

NetBeans Confirmation Dialog

Eclipse:

The Eclipse Search Dialog.  (Allows you to select scope)

Creating A Java Applet to Run in Chrome, Firefox, IE, Opera, and Safari

Just throwing together a really quick page to allow people to grab a code snipit
showing how to write html to allow an applet to be embedded in a page, such that it works in Chrome, Firefox, IE, Opera and Safari.

The following should work.
I would love comments on better ideas.

Running 47906Proxy as an Applet

This article describes running the 47906Proxy as an Applet:

Brief History of the 47906Proxy:  The proxy was developed initially as a port forwarder, or simply a forwarding proxy.  What is a forwarder proxy, and how is it used?  Three use cases are included below the 47906Proxy demonstration applet (the use cases include Tunneling, Application level Protocol Sniffing, and Network Latency Trending).  In addition a code snip-it has been included below the proxy to allow for a cut and paste usage on your website.

USE CASE 1:  

Firewall hopping.

Firewall hopping is some times called tunneling, this will allow an application to access a port on a remote machine behind a firewall, without the need or requirement to poke a hole in the firewall.  Additionally, forwarding proxies can add additional checking and validation to ensure that the individual trying to tunnel has the privileges and rights to do that.

USE CASE 2:  

Protocol Sniffing

This allows a very high level of stream sniffing, or application layer protocol sniffing.  Frequently this is much more desirable than low level packet capture sniffing.

USE CASE 3:  

Performance Monitoring

This allows for data flow rate analysis, and latency analysis at a much higher level than a packet sniffer.