lano1106 BASE v2.1 journal

[lano1106]

Stage 2, day 24:

Since yesterday, things did unblock big time. My arbitrage system is finally up an running. The estimations that I did during day 6 of stage 2 were very pessimist. I estimated that it would take over 100% CPU usage and over 1Mbps of incoming bandwidth.

Reality is very far from that estimation. I'm using less than 5% CPU and maximum bandwidth usage that I have seen is 50 Kbps.

I guess the way that I interpolated was flawed. I took possibly the busiest exchange (XBT/USD) and I multiplied its volume by the number of pairs that I would subscribe to receive their data. A lot of pairs appears to be very low volume.

Also, the 15% CPU usage that I was seeing. The huge CPU usage is mostly caused by the GUI. For now, my arbitrage prototype is command line only. This is why it is so low demanding in CPU resources.

Bottomline, this wrong estimation did make me fall into the classical premature optimization programming evil.

I work late last night. Getting close to the finish line was giving me energy to keep going until I finish. On the first run, well, it did run fine but there was a glitch in my code because it was reporting trading opportunities giving ridiculous 300000% yield.

When I saw that, I said ok. This might be tricky to find that glitch. I better try to get some rest before chasing it. Finally, early this afternoon, I squashed the last bug.

There was another glitch where all reported opportunity had a time window of 0 usec. That means that as soon as the opportunity was spotted, it was immediately disappearing. That was possibly another bug in my code. I have changed the way I measure the time window. This is important to know before I start executing the reported opportunities. How much time can I hope having to execute the trade?

Next, I did filter out silly opportunities. I was reported trade ideas that could return a dime in profit. This is ridiculous due to the inherent risk of trading. Going for it would be the equivalent of bending over in front of a steamroller to pick up a dime on the ground. Not worth it.

So my improved and fixed system has been running for the last 40 minutes without reporting any opportunity. I need to let it run at least 24h before concluding anything. In a period of about 3h this afternoon, it spotted about 20-30 opportunities. Some were looking very juicy with a yield of 6%+ but it was unconclusive because they were reported with all the glitches that the program still had this afternoon.

but now, it is very quiet. I had no idea what to expect. I wish that it would be crazy as in a casino where you hear all the slot machines going crazy all around the place 'ding-ding-ding-ding'. No, it is not what I see right now...

but hey, if my computer can generate few dollars every day effortlessly, I'm going to take it.

I think that I'll go see a movie at the theater. I want to go see 1917. I'll let my computer run and possibly when I return, I'll get some result data to crunch...

[lano1106]

Stage 2, day 32:

I have been enjoying a LOT the 1917 movie. It is an incredible well-made movie. It does a great to job to actually depict the horror of what it was to be there.

Final stage 2 day. I honestly don't know what to think about the stage 2.
On one hand, I have been diminished by the flu. I feel like I did never fully recuperated from it. I felt like my productivity did take a hit since when I got the flu. I sleep a lot and when I start to feel tired at night (earlier than usual), I go to bed instead of continuing working (could be script 'Maximum Sleep Quality' and 'Paced Working & Entrepreneurial Efforts To Steady The Ap-
proach/Drive/Motivation & Prevent Burnout' that does make me take this decision...

My marketing business income has started to decrease. I was stunned by how much money it was generating despite the minimal amount of attention that I was giving it. Well, there was some momentum that kept going and there is a limit to how long it can last alone.

OTOH:

1. my project is advancing nicely. Milestones aren't reached as fast as I would like but we are getting there. I have a tendency of underestimating the amount of work needed to reach the milestones (Or I stumble on non-essential tasks... Or it is is a mix of both reasons).

So to follow up to my last entry, I have nice signals since last weeks. The only remaining step to actually start making money is to finish the module that actually execute those signals. I was thinking that I would only need 1 day last week to achieve that and it is not up and running yet today. I'm getting closer everyday. The wait is painful, as I am nagged by all the signals as if they are telling me: See all the money you could make if your job was done that you are instead leaving on the table...

2. My debt did melt without any conscious effort from me. This one is stumbling me. It is as-if I have found some treasures under some trees outside. My debt did melt by severel thousands of dollars and I cannot explain exactly how it did happen. This is really a subconscious process.

Based on all that, stage 2 has been positive overall...

With stage 3 starting tonight, another chapter in the adventure will be starting!

[lano1106]

Stage 3, day day 4:

After 3 days of starting stage 3, I finally start feeling that I am now on high gear to to reach success.

It is funny, I'm super close to completion and start to reap the fruit of my effort. I feel like there is a part of me trying to find distractions to escape from success.

It is unbelievable that I feel like I am only a day away from completion for the last week and a half.

If I want to be fair with myself, I may have stumbled on some maybe non-essential tasks but OTOH, the amount of required work is colossal. More than I anticipated but this is my kids spring break this week and I did dropped them at their grand-parents place last Thursday.

Thursday, I was not feeling well. Among other things, I did 5 miles of rowing in the morning and I feel that I did too much and it kinda taxed my energy level for the rest of the day.

but Yesterday, I was in THE zone... I did code like a crazy monkey... code was flowing... clever and elegant solutions were coming quick to my mind... Now the goal is becoming real real close. For real this time. Like, I think it will be up and running possibly by the end of the day.

I want to go out tonight BUT this is the reward for completion. I need to reach my goal first.

[lano1106]

Stage 3, day 6:

I haven't completed the task leading me to the milestone of actually sending out an order automatically yet.

It is not because I did drag my feet. I did work many hours every day since last update... but tonight is the night... I'm 2-3 functions away from completing writing the whole thing... for real this time...

Here are some interesting stats about the project.

I wrote about 40K lines of code in the last 2 months.
My system is currently processing 1 million market updates in a little bit less than an hour.

The thing runs smooth like a baby tiger sleeping in the arms of its mother (weird analogy. I know. That is the image that came out of my mind...). It runs for days without any hiccups...

he he... Millionaire life could start as soon as tomorrow...

to be continued...

[lano1106]

Stage 3, day 7:

Guess what?

I'm not done yet. I cannot explain what is going on. My best explanation is:

1. Unexpected difficulties and challenges arise as I go.
2. When I look at the high level big picture of things. I see all the big building blocks completed. I just see the little bit of glue missing between 2 components but when I start building a test scenario and ask myself what do I need to try out this new system. I always come up with some crucial but also easy to overlook elements that are missing and that I didn't think about at first.

So the end result is: I completed very important needed things yesterday evening. Because I thought that I was close to completion and I was seeing a lot of juicy opportunities being reported by my program. I wanted to start being able to execute orders to start taking advantage of those opportunities.

I completed 3 major tasks. I had to stop when I was totally exhausted not able to think straight anymore.

Same thing today. I completed 3 other important tasks. but 3 new ones got added to the list.

I'm less confident than I was yesterday that there is a chance that I complete tonight. Best scenario would be sometime tomorrow...

Something was going on in the cryptomarkets last night. To give you an idea, opportunities usually reported have a yield in the 0.1-0.5% range. Sometime a 1.25% one will come out once in a while. Last night, dozens of 2%-3% opportunities were showing up.

I was a bit skeptic because usually, the opportunities remain valid for 100ms at best. Those 2-3% opportunities did remain valid for several seconds. Either 2 things:

1. It did happen yesterday morning. because of the crash panic, the exchange was flooded by an incredible volume of orders making it somewhat unresponsive and it was in that state too during the night.
2. This a market crash aftermath effect. Among all the asian market participants (I'm assuming that they are the majority of the crowd during the night), a lot of people were desperate to sell at any cost and there was not much courageous buyers out there...

I didn't see the trading opportunities bonanza continue during the day today...

The other thing that I want to report is extreme fatigue. It coincide a week of listening of stage 3. Last time that I felt like that was when I was running SM. The fatigue did last during the whole stage 3 and 4. As soon as I started stage 5, the fatigue went away as fast as it did come.

I feel like I'm experimenting that same type of experience. Some part of me is putting a lot of energy resisting something that is in stage 3...

Bottomline, I'm really stubborn. I'll go through it no mater what...

[lano1106]

Stage 3, day 11:

3 days ago, I did finally complete the Execution Engine module. I can send it simple orders and they are executed.

The funny aspect is that I started to test it and the various triggered events were making my code crash. I was debugging the problem. Recompiling the code and running it again. After 4-5 attempts. It finally worked. And this is at that moment that I realized the testing did cost me about $100 because even if my code was crashing in the middle of the operation, they were still going through! (it is not really lost as I purchased BTCs with it.). The minimum order size is roughly $15-20.

Concerning the results, I have learned something important. The execution timing. It is very variable. It can go as fast as 200ms or as long as 900ms. One of the thing that I haven't implemented yet can provide an execution perhaps 30ms faster. Still, considering that my arbitrage system is measuring trading opportunity that are available for about 100-200ms. This is casting a shadow of doubt on the viability of my idea... I'm NOT ready to drop it yet. I guess that I'll have to say goodbye to 100% execution success rate. This may lead me to raise the bar on how profitable an opportunity must be to do it so that successes profits outweight failure losses... hehe... This is definitely a rabbit hole... To more you scratch, the more you find challenges... but hey, I find all this super fun to work on. Next step to be able to make an assessment will be when the rubber hit the road for real...

I like what did happen yesterday. I had a list of tasks to add features and sophistication to my Execution Engine. ie like allowing its users to specify an execution strategy to modify the aggressiveness of the execution. The ultimate goal is to have active strategies. ie: Actively monitor the order books and play games with other traders with an automated algorithm.

but yesterday evening as I was contemplating the task list. I abruptly stopped and realized that despite fun and valuable this venture is, it wasn't helping to complete my #1 priority goal which is to have an up and running arbitrage system. I immediatly created a second list which is for accomplishing the most important goal.

I'm quite happy. In the past, I used to drift away for longer. It has been a source of failure. I always had a lot of good ideas. I rarely did bring them to completion. It is a lack of focus because I am curious, find everything interesting. I have a strong desire to understand everything. I guess understanding everything is a way for me to feel in better control. Well, in the job market this trait was very valuable. I could to job interview extremely confident allowing me to expose my knowledge and emanate a vibe of an expert in my field. BUT, this can be a weak point when it comes to complete projects.

idk, maybe this behavior is an unconscious sophisticated way of self-sabotaging my chance of success

So, I have steer back to the right direction and I'm heading toward completion much faster than I would before BASE.

TBH, however, I did stumble again working on a small non-crucial improvement. Ok, I forgive myself for it because, it is very small and I did reserve it for the end of the evening where my mind was starting to be tired. It was still able to execute simple improvement tasks but not important complex ones where I need 100% of my mental awareness to not make mistakes...

Finally, I did contact the service support few times by opening few tickets for asking for clarification about their API behavior. They are the most courteous service and competent technical support service that I had the chance to interact with for a very long time. but somehow I feel that the way that I am communicating with them is making them want to help me super well above what they have to contractually provide.

Covid-19 is starting to impact me. Since yesterday, the school of my kids is closed. They will stay home for at least the next 2 weeks. Same for my gf who works in a daycare center. My office is going to be crowded more than usual. So happy to already work from home. My business is minimally impacted. This is a real blessing. I was talking to a colleague who was scared of seeing his # of sales drop in the upcoming months because of covid. Funny enough, I have the opposite feeling. I'm much more optimist than him. See usually the quietest month in terms of sale is usually July because people leave in vacation and stop reading their emails. I think that with all the quarantines going around... this might be different this year. People will be bored to death locked in their house with no sport at all at the TV. They will be more than willing to sit down in front of a lengthy VSL... Is my enthusiasm caused by BASE? maybe...

[lano1106]

Stage 3, day 16:

My productivity is impacted by the CoVid-19 pandemy. I have been locked down for almost a week now with the rest of my family.

I haven't not finished my work on my arbitrage system but it is progressing every day. A new pattern that I have develop and I think that BASE is responsible for that. It is that when I face a challenge that appears complex and insurmountable, I just look at it. I have some kind of blurry image of how to resolve the situation.

I go to bed and usually the next morning, I wake up with a clear picture on how to proceed to reach a solution. I really love this way of proceeding and how smooth it is.

[lano1106]

Stage 3, day 19:

This starts to be a joke. Basic execution engine functionality is completed. I did meet few unexpected hurdles down the road.

2 of them are:
1. API documentation specify 2 optional fields allowing postpone order publication or set an expiry time for the order. Those functionalities simply didn't work and/or were only partially implemented. TBH, and I wasn't aware of that fact, the WebSocket API did only allow to place and cancel orders only since Feb 18 last month. Since the start of the project 3+ months ago, these 2 calls were in the documentation and I did simply assume that they were working. I'm lucky. They just happen to be available just in time when I'm ready to start using them. So the unexpected cost of that discovery is that I had to implement those missing functionalities in my client. Not rocket science but it cannot be done in few minutes neither. (I think that thing alone took about 3 days. As a side bonus, this has given me the opportunity to improve further the existing code as I spotted improvement opportunities as I was reworking the code for the new functionalities...)
2. I have discovered subtlety where the exchange various assets precision/decimals isn't the same as what can be stored in a double variable. Not sure exactly how they round the various values but I'm pretty sure that it is safe to assume that they pocket all the rounded fractions. So I truncate when an amount goes in my account and round the value to the next integer value above when I'm paying. Bottom line, they have an API call that gives you a JSON output with all the decimals for the 40+ supported assets. I need to write code to get that JSON file, process it and use the result in many locations in my code.

On top of that, now that I have a simple working execution engine, I need to add a new layer in it where it can execute a trade plan which consist of a sequence of orders. Again, I'm going to estimate that task to few hours so hopefully, it is going to be done either tonight or sometime tomorrow...

Building this thing is like trying to break a concrete dam with an ice pick. It looks like an impossible task but as soon as a hole is pierced, things should precipitate real quick eventually...

In terms of metrics... I'm now at ~45K line of code. Last time that I did measured that metric, it was on stage 3, day 6. So 5K lines in 13 days... About 400 lines/day... I have got more productive peaks in the past but I guess when you get close to the end and have few remaining though nuts to crack to complete, it is normal to experience a small slowdown...

but things still go in the right direction...

[lano1106]

Stage 3, day 22:

I have a bunch of unordered thoughts. Here they are in no special order:

1. For the, idk how many times, I have underestimated the amount of work remaining. So that when I said that I would be done 4 days ago isn't yet. One day, it is going to be true because I keep progressing.
2. I did a mistake when I did count # of lines of code by forgetting to include a directory. It is now more close to 50K lines of code instead of 40K. In the last 4 days, I must have written about 1K lines. Not much more than that.
3. I have this strange feeling that some unconscious process may be slowing down my progress as some sort of way to self-sabotage. I feel a little bit less focused than when I started 3 months ago. To be fair with myself, I need to account for all the Covid madness going around. That must have an impact.
4. When I'm seeing around all the people around suffering from job loss and financial trouble, I realize that I must have some sort of 6th sense about what to do for my financial safety. 3-4 years when I did retire the job market to launch my digital marketing business, I had no clue about what calamity would fall on us but it turns out that by having done that transition has kinda protected my income in these hard times. I'm wary a lot about what the government will do next. For now, it is handing cash to everyone for "free" but I forecast that it won't stay "free" for long and a lot of not so fun strings will be attached to the payment in a not so far future if/when we are 18 months+ into the covid thing that does't want to go away. I prefer to stay away from government "free" money.
5. Another manifestation of that special 6th sense. Back in November, I had again no clue about what was to come when I decide to make another professional transition to crypto trading. Now that paper money printing press are printing money like crazy and governments around the world start talking about putting in place digital money because cash is "dirty", it seems again that I'm at the right place at the right time...
6. The fatigue that I reported at the start of the stage is gone but, it has been a month since I did catch this nasty flu. The fatigue could have been induced by my flu. And also, I kinda have adopted some sort of quarantine beat. Since morning parental obligations have kinda soften a bit, I sleeping easily and effortlessly a good 10h per night which I find is a lot of sleep.
7. Out of my project tasks list, I did find some sort of shortcut to speed-up completion by weeding out a feature that I was considering mandatory and I have somehow found a way to ease the requirement.

Again, I think that I'll be good to have something trading for real tomorrow... this for real... Anyway, I'll report soon how it goes.

Good night

[lano1106]

Stage 3, day 23:

Almost done. 2 mini baby tasks remaining before its first flight.

I think that I'm good to finish this before dinner and make the first system test later tonight!

edit: Done!!!!

Finally, the initial ARB system prototype is up and running.

Now, I just need to wait for it to detect trading opportunities. It could happen soon or it could take hours before it happens. Those opportunities happens in sudden bursts and clusters.

I'm playing it safe for the first run to avoid potential overlooked bug to empty my account while I sleep. I'm limiting to 1 trade. The goal here is just to see how the code behave and catch any remaining bugs, if any.

Because of where I run the program, it is a bit slower than what it could be and since it is only the first version of my Execution Engine, it is a bit simple and dumb. Next iteration (maybe tonight or tomorrow) it is going to be more dynamic and this will compensate for the slow reactions.

Iteration #3 could be: dynamic + fast execution for very good results to come I hope...

I'm going to take few minutes to celebrate this important milestone that I have been working for so hard since the start of the year...

[lano1106]

Stage 3, day 24:

My first ARB trade is positive.

As predicted, because of the static nature of the current execution engine version, when it did reach trade #2 (1 second after the start of execution), the detected entry justifying the whole thing was already gone. My order has been stuck for about 16h in the book. This is when I did discover how the PAXG/ETH market was lacking liquidity. You don't want to be trapped there when you want to move fast...

I have been working on making my execution engine more dynamic today. An execution shouldn't last more than 2-3 seconds. Not days!

I have the big picture of the extra data structures needed to add dynamism but the details are still a bit blurry. I need to find the best structures that are meeting the various requirements to support all the operations that will be performed on those.

In laymen's term. I'm kinda stuck in front of blank page. I'm starting to write some code hoping that as I go, things will become clearer. Also if past is warrant of the future. I could as well wake tomorrow with a clear solution as my subconscious will have find out a solution while I sleep (very cool phenomenon that I'm learning to use).

Anyway, by some miraculous events, the markets went in the right directions for me and I did exit the trade with a profit of $0.28!

It doesn't seem much but when everything will work smoothly, it has the potential to repeat the same process hundreds of time per day.

Accounting is way off... Numbers that I compute are way off from what is reported by the exchange but those things will be polished. The core has worked flawlessly. That is what is important.

[lano1106]

Stage 3, day 25:

data structures details for enhanced execution engine dynamism found this morning!

More to come later in this journal entry if more good stuff happens...

[lano1106]

Stage 3, day 30:

Data structures went perfect. Part of the solution was to use a Finite State Machine pattern.

I did develop several execution strategies. about half a dozen of them. I did roll out some of the more complex one that I'll need in the next step.

Something that I did realize while I was testing. it is that those execution strategies works so well that you can practically rely on them to have an execution that is almost as fast as a market order but with the maker fee you receive with a limit order...

One analogy that I could make. Those strategies in the trading worlds is what the 12 different words that Inuits use to describe snow.

It is really fun to test it live. In fact, my test has been profitable. As I was buying low and selling high BTC. That wasn't really the goal. I simply wanted to test the execution but it came naturally that I could start make a small profit while I was testing. I simply looked at the price and the type of order to use for my testing *and* making profit was obvious. I didn't make a fortune since I was testing with minimum accepted trading value (about $15) but it is the mindset that I now appear to have that did please me the most.

At each testing iteration, I was also discovering small shortcomings that I didn't notice at first. I fixed also a bunch of rounding errors.

So that now, I have a system that works really really well. At this stage, it has become really a game. A very fun one...

[lano1106]

Stage 3, day 31:

Here is the goal to reach before end of stage 3.

Have the arbitrage system up and running with the new dynamic execution engine.

Pretty achievable goal as the only remaining task is to create one last execution strategy for arbitrage which will be some sort of aggregation of some of the strategies just created.

I feel like I could complete this afternoon but lets keep some buffer for surprises.

Next, either tonight or tomorrow, I'll do some sort stage 3 summary. With the Covid thing going on, this has been a once in a lifetime stage... I'll also take a look about what to expect about the upcoming stage 4.