lano1106 BASE v2.1 journal

[lano1106]

Stage 1, day 31:

There was another $100 drop in Bitcoin price in less than 4 minutes this morning. My trend following strategy has been way too slow to pick it up...

BUT, I'm starting to see a pattern and understand the phenomenon. I'm pretty sure that I can predict it when it is about to happen to position myself to make those 4 minutes the most profitable 4 minutes of the day... If I'm right, once confirmed, I could add some leverage to the position to really make out like thief....

[Yous]

Any indicator in specific?

[lano1106]

No indicator in specific. Simply a different insight into trading that I got from looking at those damn charts for so long. Thinking about the mechanics of the exchange, at some point, you start understanding what is causing the moves.

I have always been skeptic about the various support/resistance lines and trading channels theories. That have always sounded like hubris to me and in fact markets could really be totally random for all we know. BUT, if enough people believe and practice those technical analysis theories, they become like a self fulfilling prophecy.

I'm not sure if the development of those techniques have been accidental or seeded in the conventional wisdom for the benefit of few people who figure out that training the herd on how to react, they could harness a system that would otherwise be chaotic and unpredictable.

If a sufficient number of people set a 'Take profit' stop on their order on a major resistance line price. Once that price is reached, all the 'take profit' stops are triggered and make quite a drop in the price. This drop, if big enough, then creates a chain reaction that triggers all the 'stop loss' orders. And THAT makes a big hole.

Since I saw it happening yesterday, it did happen at least 3 others times. Since it is not a 100% sure thing. You cannot just program an order to buy or sell at a certain level and forget about it. You need to monitor the market either manually or with software automation to spot early sign of what is happening. The opportunity window is very small. 3-4 minutes max because after, it is over...

[Yous]

I agree with you in that there is a system that a few use to win money, and those big suddenly movements is to make the stop loss being executed. 

The indicators and the news are information and explanations about the past. I do it with out stop loss and indicators, but sometimes I take a look for having a reference.

My problem is that I don't know how to simulate, but I think that a system base in the tendency with a more less random entry with the stop loss (even if you don't put the order before) and the take profit just being the double could be profitable. Simple example, random entry long in bitcoin in 9.000, stop loss in 8.800 and take profit in 9.400. If you win even 40% (I didn't do the numbers im writing with the mobile) you win. If you know how to simulate and try it please let me know it.

And I don't know if you do future's or cfd but the advantage is the you can go long or short, more possibilities...

[lano1106]

Stage 2, day 1:

I love stage transition. I don't know how to explain it but when I start a new stage, I always feel some magic like feeling. I feel uplifted and my mood and energy in the roof.

I got a tough day... I had hard time writing code for my new arbitrage module. I never do that but seeing that I was in some kind of unproductive slump, I went to the gym at lunch time instead of going at the end of the afternoon. The reasoning was that doing some physical exercise could have a positive impact once I return back home...

No, it didn't help much... but the lack of productivity was only on the surface... It is quite a complex piece of code that I'm writing... Details on how things are going to work out aren't 100% clear out of the blue... but under the surface my mind is thinking about the problem and is busy mapping out an elegant solution.

As a side note, I felt that my socializing skills were on steroids. It was easier than usual to gain rapport with others while I was at the gym...

As I'm listening stage 2 for the first evening... now ideas and software design are flowing out very smoothly. The trading guy from whom I've share his video. Maybe I could manage to meet him and check if I could offer him an association. Maybe his trading skills along with my software development skills put together could create the next Renaissance like hedge fund... Maybe this idea has been suggested by stage 2.

[lano1106]

Stage 2, day 1 (morning):

I had a flash yesterday evening. Back during winter 2015, steam has just started to be offered on Linux.

One major thing that was missing was the option to broadcast/record gameplay. I did some research and I have found a nice project that was hijacking openGL and ALSA lib to intercept the raw audio/video data to record it in a file.

It was working fine but it was creating monstruous large files. I got the idea to improve the project by piping its output to FFMPEG. This has worked superbly. It was encoding decent size MP4 files in real-time without impacting game framerate at all.

but the point of this story is that this small project have made me busy for 3 months and I was listening the 3-4 Daft punk albums I have in loop non-stop as I was working. That music had a sort of hypnotic effect on me, the music energy kept me in a very alert state. I was in the coding zone. One of my best productive coding period.... This has given me the idea to relisten the same music from that period to retrieve the same productivity state that I had succeed creating back then...

[lano1106]

I wrote a small piece of code that traverse the pair graph found on page 3. I wanted to find out all the possible cycle paths that starts and ends from/to XBT.

Visually, you can spot few trivial ones. For instance XBT/USD -> ETH/USD -> ETH/XBT and few more that have 3 transitions...

but I clearly under evaluated the number of possible chains. I was at over 12 millions possible combinations when my system did run out of memory and my program crashed.

I spotted a glitch where few invalid cycles were added. I call those noops. One such noop useless cycle is: XBT/USD -> XBT/USD. I have remove those but there is only 39 of those so it is not a big part of the 12 millions combinations that I have found.

there is 2 things that I'm going to do:
1. I'll optimize memory usage of that code. When I ran out of memory the code was processing ZEC and they are processed in alphabetical order so I was almost done. To fulfill my curiosity, I want to know the exact number of cycles in that graph around XBT.
2. I'll certainly have to limit the length of a cycle chain. The longest that I have seen is 11 such as:
ZEC/XBT -> ZEC/USD -> XTZ/USD -> XTZ/EUR -> XRP/EUR -> XRP/JPY -> ETH/JPY -> ETH/USDC -> USDC/USDT -> USDT/CAD -> XBT/CAD

That length is possibly too long to be useful for anything. Plus if I have millions of combination to check every time a new best quote comes in, I won't be able to keep the system operate in real time.

On a different note, I went to do small errand with my family. I was annoyed that doing them was slow and members were distracted by things they saw on our way instead of staying focussed on the list of items to buy. I was feeling a strong drive to complete these tasks fast! It is first time that I notice such a drive in me...

[lano1106]

I succeeded having all the cycles that I was looking for.

The exact answer is 12,196,648 cycles.

It took about 2.3G of RAM to store all the data related to those cycles. It is about a little bit less than 200 bytes per cycle.

On one hand, I wouldn't worry that much if I had a computer with 128GB of RAM (I only have 12GB) but 200 bytes/cycle is a lot of memory for what I am storing. I better spend some time to optimize memory usage of this beast. Having such a big memory footprint is bad for having a good cache hit.

What I would find a reasonable memory usage is 50 bytes/cycle. And I suspect that it isn't really my data that is using all that memory but a lot of memory is wasted due to millions of small allocation/deallocation...

Update: I just managed to reduce the memory usage to about 140 bytes/cycle for a total of 1.7 GB of RAM for the 12M+ total cycles.

From now on, I'll limit the chain length to 3 and once the concept is proven to work, I'll slowly ramp up the chain length...

Update #2:

By limiting the cycle length, I got interesting numbers:

Max length - # of cycles
3 - 188
4 - 3020
5 - 16190
6 - 152,062

With a length of 6, this requires up to few thousands cycle value recalculation... I don't think this can be done real-time with my current hardware... If this turns out to be a profitable venue... I'll explore doing it with CUDA or OpenCL...

[lano1106]

Stage 2, day 3:

Today, I feel extreme fatigue. I feel like I need a lot of sleep. I remember that I felt like that in the first few days of Stage 1 and this feeling eventually went away. Or I might be fighting a small infection as my older daughter has one...

Beside, that small annoyance, here is what is interesting about my BASE stage 2 journey so far.

1. I did notice that when I was sharing my goal with others in my entourage, I'm used that it is received with some level of skepticism. Sometimes a lot and I'm being called a dreamer. I'm told that what I want to achieve is impossible and that I should settle for much less than that... Sometimes there is just a little bit of skepticism... it doesn't bother me at all but since the last few days, I feel like when I talk about my project, people are buying in and I succeed in convincing my audience into the feasibility of the plan.

That is a very welcome change. I enjoy it.

2. My shift that made me put the trend following strategy on hold to explore the arbitrage strategy may be in accordance to stage 2 script: "Only Compete Where Necessary & Use The Competition Or Lack Thereof To Your Advantage".

I feel like there is possibly less competition in the arbitrage scene than there is in the trend following arena. This of course has to be validated but I'm currently very excited to the idea of discovering an almost untapped field of opportunities.

If there is already competition in crypto arbitrage, then the name of the game is speed and scalability.

I'm maybe thinking too far ahead but right now, I have a pretty compact algorithm to calculate the value of a cycle... but it is a loop... and it contains evaluating conditional expressions...

If I had to modify that part of the code, I would need to sit down with some paper and a pen and figure out a way to do it with matrices.

If that was feasible... That would open up a lot of options. For one, there are specialized libraries that are very optimized for matrix operations... There is hardware (GPU) that can handle matrices with millions of elements in them and process them as fast as a CPU can do a simple arithmetic operation... but my linear algebra skills are very rusty.... I did prepare myself a small stack of linear algebra books from my university years in my bookshelf in preparation of having to go in that direction...

[lano1106]

Stage 2, day 4:

The low energy state is persisting. I went to the gym yesterday afternoon. My muscles are sore today. It is either the workout or a symptom of a cold or a flu.

No matter what, I'm going to kick myself to move forward the project as I am getting close to a working arbitrage proof of concept prototype...

[lano1106]

Stage 2, day 6:

Arbitrage code is done. Not tested yet. I had to revisit a communication library that is a building block of my system first.

The code is very well done but documentation is poor. So to figure out anything about how things are done, the only way is to roll your sleeves in dive in the code.

The purpose of revisiting this part is that enabling the arbitrage module will multiply the input data amount by 131. Current bitrate is a bit less than 10 kbps. So we are talking about 1.3 Mbps. Not too big but I'm concerned about its processing time. By just writing those numbers, I think that something is going to blow when I start up the arbitrage module. I currently use 15% CPU. My current CPU won't be able to keep up with a 131x bigger load... I guess hardware upgrade is now just around the corner. TBH, my current system is roughly 10 years old so this is very venerable age for a computer system... I shouldn't be ashamed thinking replacing it...

but anyway, before I realize that detail, the goal was:

1. Figure out how to integrate libev with the other communication lib.
2. Make sure that TCP parameters were well tweaked to support the increased volume
3. While, I'm at it, knowing that the very next step will be to have a client connecting to 2 different services using 1 thread per service, I better figure out how this is going to work while the not so well documented library code is fresh in my mind...

It took about 1.5 day to complete those 3 tasks. I have a clear view in my mind how #3 is going to be done in my own codebase refactoring (either tonight or tomorrow).

This has been a bit painful and stressful... Reverse engineering someone else code isn't easy. Plus, on the surface, not a lot of code is written during that time. That leaves the impression of stalling but it isn't the case. Collecting information is an important step in creating nice solution to complex problems.

Beside, I thought that I knew it all when it comes to linux system programming and I have just discovered 2 curious and useful subsystems that I have never heard about during my investigation. Not sub related but for those who like that type of details, it is eventfd and signalfd! Discovering those 2 little gems has been fun and enjoyable.

[lano1106]

Stage 2, day 7:

Devil is in the details. I did stumble into a bunch of problems due to my library apparently uncommon usage. I took me few unexpected extra hours to get rid of them.

As an extra bonus for all my efforts, my software is now using the Linux AIO event backend which is apparently the best possible option when working for you...

[lano1106]

Stage 2, day 16:

I feel like I have fallen into a rabbit hole during the last week. I saw an opportunity to contribute to my FOSS communication library and help others. This sub-goal did distract me away from MY main goal and it did turned out to be mostly a waste of time as the project maintainer isn't very receptive about receiving help, very argumentative, wants to have the last word all the time and kinda makes you feel like an idiot because he doesn't understand what you say.

imho, if you want to be at the head of a successful project beyond what is humanly possible to do alone, you need to improve skills to attract and keep people around to support you... I got the perfect example of what NOT to do to achieve that.

So overall, it was a bad idea.

And, I have fallen sick. A terrible flu. Because an old chinese elder man did bump into me few days before and with all the media attention on the Cov-19 pandemy, I did imagine the worse... I have been in bed, with fever, no appetite, totally knocked out and absolutely unproductive for the whole week...

what is done is done... now I feel better, it's time get back on the right track and continue what needs to be done to succeed.

[lano1106]

Stage 2, day 22:

I think that I am now fully healed from my flu. It did disturb my normal routine for the last 2 weeks. I was thinking returning to the gym yesterday for the first time in 2 weeks but a small headache did make change my plan at the last minute.

My project is progressing well. The huge road block that was in front of me has been conquered and I'm about to move on to the next step.

As a very interesting side note, since the beginning of the year (and BASE listening), my personal and corporation debt have melted in half despite my new venture taking most of my day to day attention. It feels almost like magic considering that I have struggling achieving this for years and being my #1 priority....