Creating a COSA virtual machine

Discuss the creation of a COSA virtual machine.

Re: Creating a COSA virtual machine

Postby sparky » Sun Sep 19, 2010 4:41 pm

Here is my "final" design:

I have made the assumption that developers will be using an established operating system (Windows, MacOS, Linux, DOS) for the foreseeable future. That's not to say this design won't run on bare metal, but someone would have to implement input, output, drivers...and that's not me.

User interface layer
-----------------------
The user interface layer provides an interface between the OS and the COSA processor. On Windows, applications are either Console-based, Forms-based, or DirectX (WPF)-based. I've chosen to deal with the first type, Console-based. Other implementations would have to convert OS input to a COSA input and COSA output to the OS output. With that, I have a CosaConsole class that takes user input (in this case, keystrokes)and passes it along to the internal process. Conversely, the internal process contains an output buffer that the CosaConsole reads every tick and displays the results (in this case, text only).

I have only implemented a "debugger" application which is infinitely more useful when developing the core and testing new features. Realistically, I would have a second console application that runs the internal processor without interruption. I envision a separate thread that runs alongside the internal processor that captures keystrokes and puts them in the internal process's input buffer. Since I'm debugging in a console, I limited the number of processes to run to one (easier to test).

In summary: a console interface layer captures keystrokes from the user and puts them in the internal process's input buffer. Any output that the internal process wants to display is put in an output buffer, where the console interface picks up at the end of each cycle and displays.

Internal process
-------------------
I have implemented a straight sensory processor, but I believe that for applications of any complexity, this has got to be broken down into sub-components. An internal process contains the following elements, which are defined on process initialization/loaded into memory:

* List of all available effectors, searchable by name
* List of all available sensors, searchable by name
* An in-progress effector queue
* A completed effector queue
* List of tests to run at the conclusion of each tick

Name: What's in a name? It could be a memory address. I've chosen to make it readable so when I reference it in my sensors and tests, I know which one I'm dealing with.

Effector and sensor lists: I believe that all effectors and effectors should be predefined up front, just like all variables should be defined from the beginning. It also makes it easier for sensors (which use effectors) and tests (uses sensors and effectors) to point to references than try to manage them ad-hoc.

The in-progress and completed effector queues are self-evident: effectors to be run are put in the in-progress queue and moved to the completed queue when run. While I've implemented a linear-loop for effectors, the order that the effector runs should not be considered to be in any particular order.

List of tests: once the sensors determine what has taken place, a test determines the sensor state and acts accordingly by turning on (start) or off (stop) effectors. Again, the reason for the master list of effectors.

Internal sensory process: Start
-------------------------------------
1) The implemented application specifies which effectors to turn on initially.
2) The internal processor adds those effectors to the in-progress queue.
3) The internal processor turns on all sensors.

Internal sensory process: Execute
---------------------------------------
1) Clear the completed queue from the previous tick. (I want to keep it around for debugging purposes, which happens at the end of a tick).
2) Clear the changed flag on data values (which may have gotten set from the previous run)
3) Execute each effector in the in-progress queue and copies it to the completed queue.
4) Execute all active sensors (this sets some internal flags inside the sensor: ZF (zero), SF (less than), OF (overflow))
5) Execute all tests (based on the sensor's internal flags and the state to evaluate, turn on or off effectors)
6) Clear the in-progress queue and add any effector that is started.

NOTE: once an effector is turned on, it is never turned off unless a test explicitly does so. It will execute on every tick.

At this point the debugger can view the status of the internal process: what effectors ran, what will run, and assigned data values.

Internal sensory process: Stop
-----------------------------------
1) Turns off all effectors and sensors, so when restarted it will be in a consistent state.
2) Clears the in-progress queue.

The basic building blocks
-----------------------------
* Data value: name, changed flag, value (depending on the type). I implemented an integer, byte, byte array, and collection (name/data value pairs) type. The internal key input uses a specialized collection instance. Values can be created with or without an initial value.

* Effector: start/stop operations simply change the started flag (does this run on the next tick or not?). Execute runs a primitive operation: set the specified data value to a new value, increment the data value, write text to the process's output buffer. Every effector knows the main process, so it can get the reference data value on initialization.

* Sensor: start/stop operations simply change the started flag (does this run on the next tick or not?). Execute takes data values and sets the internal flags based on the what it is sensing. For example, my comparison sensor takes two data values and compares them for equivalency, less than or greater than, like the CMP opcode.

* Test: start/stop operations simply change the started flag (does this run on the next tick or not?). Execute does an equivalency check. For example, if a test requires two values to be equal, the test looks at the sensor's ZF flag. If the test condition is satisfied, the test turns on or off effectors.

Conclusion
------------
Below is the binary for my implementation. I compiled it for .NET 2.0. The application and DLL file must be in the same folder. I have also attached two sample files: a hello world and a hello world that loops 5 times before declaring it's done. I do not actually read these files, since I'll give the community an opportunity to comment. I am willing to make minor changes, but structurally this is it. I've done six iterations and need to move on.

Using the debugger is relatively straightforward. The 'help' command will give you the available commands. To open a file, enter either 'open sensorhello.xml' or 'open sensorloop.xml'.

If you want the source code, please PM me.
Attachments
CosaConsole.zip
Main executable, supporting library, sample files
(12.97 KiB) Downloaded 697 times
sparky
 
Posts: 22
Joined: Sat May 22, 2010 12:34 pm

Re: Creating a COSA virtual machine

Postby sparky » Tue Sep 21, 2010 5:30 pm

Update to above. I figured out how to efficiently tie components together, using the synapse model.

So in a nutshell:
1) A UI layer contains a process, which translates inputs to a platform-neutral format for the process and takes output from the process and converts it into a UI-specific format.
2) A process contains at least one components and may have one or more synapses. Synapses tie two components together and are either activated or deactivated.
3) A component contains at least one effector, and may have one or more data values, sensors and sensor tests. Effectors perform actions on data values, sensors detect state based on the data values, and sensor tests turn on or off effectors based on the sensor results. Sensor tests also activate synapses.

On process start:
* Load components, which initialize effectors, sensors, and sensor tests
* Start the initial component(s)
* Put the started components in the "in progress" queue.

On process execution (once per tick):
* Clear the completed queue
* Execute each component in the "In progress" queue
- Start other components based on synapse results, turn off synapse
- Add to completed queue
* Clear the "in progress" queue
* Add any components that are started to the "in progress" queue

On process stop:
* Unload components

On component start:
* Determine which effector(s) are initially started
* Put started effectors in the "in progress" queue
* Start all sensors

On component execute:
* Clear dirty flag on values
* Execute each effector in the "in progress" queue, and put in the "completed" queue
* Run any active sensors
* Run all tests -- if a test refers to a sensor that is not active, it does not perform the test
* Clear the "in progress" queue
* Add any components that are started to the "in progress" queue
* If there are no effectors to run and all sensors are turned off, stop the component.

On component stop:
* Clear the in progress queue
* Turn off all sensors

I also added the ability to define how many ticks an effector should run before stopping, which makes the file simpler. I've attached my two sample XML files below with these changes.
Attachments
SensorLoop.zip
(989 Bytes) Downloaded 678 times
sparky
 
Posts: 22
Joined: Sat May 22, 2010 12:34 pm

Re: Creating a COSA virtual machine

Postby harry666t » Sat Sep 25, 2010 12:25 pm

Hi Sparky. I've got a lot to comment, but so little time :P

Session application: contains a collection of running processes, with the ability to add processes while running. Each process can self-terminate or to be forcibly terminated by the session application.


Dunno. The definition of a process in a classic OS is: a kernel thread with the so-called user context. There are no threads in COSA, and I believe that the classic separation "kernelspace / userspace" is too simple to provide decent security.

And one more thing. I've said it on many occassions and I feel like it needs to be said once again: there are no reasons for applications to ever terminate. Just like there are no reasons for your spoon or for your door to terminate.

For example, a virus scanner performs a scan when a file is accessed. But it can also do a full scan every day automatically.


The mere presence of a technology like a virus scanner implies an insecure system. Why bother with viruses when we can have ultimate security through the means of capabilities: http://en.wikipedia.org/wiki/Capability-based_security

The CBS model has been mathematically proven to be secure, so while we're redoing software development from scratch, why not use this for our advantage?

An auto-save feature runs periodically.


I think that the state of the document on the screen should always be reflected by the state on the hard disk. The moment you finish typing is the moment you've got your data safe on the disk (or on a remote server). If a crash ever happens, you shouldn't lose more than the last 10 seconds of your work. If you want a particular version of the document to be archived, just provide a "snapshot" function. Some version control could be integrated with the storage, ZFS-style.

I'm not a game developer, but they are certainly more complex than a business application.


I do some simple game development for fun. Games are basically soft-realtime applications (like other multimedia software). Most games revolve around an event loop (get user input, process physics and logic, draw stuff on screen, at least thirty times a second). I guess COSA could do really well here (it's basically all about events).

A windowing environment aids a user in accessing and manipulating the operating system's process list,and providing a common layer to the loaded devices (like OpenGL or DirectX). It doesn't provide any other additional features that the operating system doesn't already do.


We've had some discussion on GUIs in other forum threads. We've agreed that Jef Raskin's ZUI is a pretty nice concept, and that it'd work well both as a development environment and as a "regular desktop".

Server applications
Business applications
Game applications


Well, how about:
* desktop utilities (calculators, calendars, etc)
* managing information (photos, music, bookmarks, contacts)
* editors (word processors, image editors, sound recorders & mixers)
* communication (email, chat, file sharing, microblogging)
* multimedia playback (music, video)
* system utilities (software upgrade, hard drive utils, etc)
* personal settings (appearance, keyboard layout, behavior of UI)
* remote system management
* number crunching (simulations, calculations)
* soft- and hard-realtime applications (DSP, controlling a robot)
* making all of it easy to work on and easy to work with
* integrating all of it seamlessly, each with everything?


3) Effectors run in an indeterminate order. The consequence is that an effector that relies on another effector to have run cannot occur in the same cycle.


True.

That's not to say this design won't run on bare metal, but someone would have to implement input, output, drivers...and that's not me.


Maybe Casper :) he worked on a very cool real-world OS.

Maybe we could use Linux (the kernel) as a base (lots of drivers and cool kernel APIs), integrate it with the CVM, and supply the rest on top of it?

On Windows, applications are either Console-based, Forms-based, or DirectX (WPF)-based.


How about using cross-platform libraries? GTK and Qt are both fairly cross-platform widget toolkits, while still being fast, powerful and good-looking. OpenGL and SDL also work well on all three major operating systems. Using Windows-specific solutions would do nothing but double the work.

Also let's not forget ZUI. It would most probably need to be implemented directly on top of SDL, X11, OpenGL, GDI, or some other very low-level layer.

Other implementations would have to convert OS input to a COSA input and COSA output to the OS output.


Sure. This is called binding. So, before we go anywhere further, I propose that we discuss some kind of universal binding mechanism for exposing foreign APIs to COSA programs.

List of tests: once the sensors determine what has taken place, a test determines the sensor state and acts accordingly by turning on (start) or off (stop) effectors. Again, the reason for the master list of effectors.


Can you elaborate on sensors & tests? I think I don't fully understand the problem to which they are the supposed solution.

Sensors (or tests, or whatever) shouldn't make decisions on turning anything on or off. They should just send signals. "acts accordingly by turning on (start) or off (stop) effectors" - this sounds like a jump instruction that inspects its target memory address...

Remember that sensors and effectors are not the only kinds of cells known to man, there are also logic sequence detectors (LSDs, haha), which most often have a variety of special input connectors.

On process start:
* Load components, which initialize effectors, sensors, and sensor tests
* Start the initial component(s)
* Put the started components in the "in progress" queue.

[snip...]

On component stop:
* Clear the in progress queue
* Turn off all sensors


It came to me that this design is broken.

Sensors can watch over data that sits in other components. Effectors also can act upon "foregin" data. If you run components separately, one by one, in any order, you effectively let sensors randomly run before or after effectors, thus creating one big race condition.

Of course you can forbid data sharing between components. But that would seriously limit the potential of your implementation.


If you want the source code, please PM me.


This ain't how open source works :)

By the way, you provide LOTS of updates, submit lots of code, etc. It would be much easier to follow your progress if you used some kind of VCS. Your design seems to be evolving pretty quickly, and you make lots of interesting decisions. I want diffs :D
harry666t
 
Posts: 48
Joined: Thu Mar 11, 2010 7:38 am

Re: Creating a COSA virtual machine

Postby sparky » Tue Sep 28, 2010 1:19 am

I was happy that someone finally responded, then disappointed as what I got.

harry666t wrote:And one more thing. I've said it on many occassions and I feel like it needs to be said once again: there are no reasons for applications to ever terminate. Just like there are no reasons for your spoon or for your door to terminate.


And why is that? In the complementary theory, something must be born if it is going to die. Are you suggesting that applications stay in memory in perpetuity? That everything is loaded when the OS starts? That's a big waste of memory. Unless you're thinking that applications self-hibernate when stopped, saving themselves to disk when not in use. If so, then how does it get in a hibernated state to begin with? When the component is released?

For example, I have a sump pump that runs fairly regularly. It starts when the water level hits a certain point then stops when it has lowered the water to certain level. But if the power goes out, it doesn't turn on. This is my definition of termination. It's not going to start unless I get my generator going or the power company fixes their problem.

harry666t wrote:The mere presence of a technology like a virus scanner implies an insecure system. Why bother with viruses when we can have ultimate security through the means of capabilities: http://en.wikipedia.org/wiki/Capability-based_security


You missed the whole point. I'm not intending to write a virus scanner. That is an example of a component that runs periodically without user intervention, just like the auto-save feature. Unless you were trying to tell someone "don't do it!"

harry666t wrote:Well, how about:
* desktop utilities (calculators, calendars, etc)
* managing information (photos, music, bookmarks, contacts)
* editors (word processors, image editors, sound recorders & mixers)
* communication (email, chat, file sharing, microblogging)
* multimedia playback (music, video)
* system utilities (software upgrade, hard drive utils, etc)
* personal settings (appearance, keyboard layout, behavior of UI)
* remote system management
* number crunching (simulations, calculations)
* soft- and hard-realtime applications (DSP, controlling a robot)
* making all of it easy to work on and easy to work with
* integrating all of it seamlessly, each with everything?


Again, not sure what you're trying to say.

sparky wrote:Server applications:
Servers have only two types of activities: user-based or temporal-based. For example, a web server does nothing until a request instance is placed in the server's queue by a web browser on behalf of a user. That request is passed off to a processor asynchronously, that ultimately leads to a response instance passed back to the calling browser. If no more requests are in the pipeline, it waits. Some servers have a temporal component: every x minutes, perform a task. For example, a virus scanner performs a scan when a file is accessed. But it can also do a full scan every day automatically. Server applications can be customized by changing their operating properties only, and accept a single, predefined type of input: for example, an FTP, TELNET, or HTTP server is sent a command from a limited set of lexicon.

Business applications:
Business applications are like server applications, in that they also have only two types of activities: user-based or temporal-based. But they also have a user interface that allows for changing of content besides settings. For example, a user has no control over how a virus scanner scans the files, only when and which files to scan. A document application, on the other hand, allows a user to customize every document that is produced. An auto-save feature runs periodically. Business applications can accept multiple forms of input: keyboard, mouse, streaming data, and so on.

Game applications:
Game applications accept user and temporal activities. But they also have processes that run all the time, to simulate the environment that the player is located in. Game applications utilize a full range of input and outputs. I'm not a game developer, but they are certainly more complex than a business application.


* desktop utilities (calculators, calendars, etc) - user/temporal based (business)
* managing information (photos, music, bookmarks, contacts) - user/temporal based (business)
* editors (word processors, image editors, sound recorders & mixers) - user/temporal based (business)
* communication (email, chat, file sharing, microblogging) - user/temporal/running processes (game)
* multimedia playback (music, video) - user/temporal based (business)
* system utilities (software upgrade, hard drive utils, etc) - temporal based, no ui (server)
* personal settings (appearance, keyboard layout, behavior of UI) - user based, ui (business)
* remote system management - user based, ui (business)
* number crunching (simulations, calculations) - temporal based, no ui (server)
* soft- and hard-realtime applications (DSP, controlling a robot) - user/temporal based, ui (business)

Perhaps you are confused over the category names. None of your examples fall outside of my three containers.

harry666t wrote:Maybe we could use Linux (the kernel) as a base (lots of drivers and cool kernel APIs), integrate it with the CVM, and supply the rest on top of it?


I found an open-source OS that they ported X11 to (the successor to the MIT). But the question is do you want a massive monolithic kernel or a microkernel? I've been thinking about using DOS has a base platform (FreeDOS is open-source).

harry666t wrote:How about using cross-platform libraries? GTK and Qt are both fairly cross-platform widget toolkits, while still being fast, powerful and good-looking. OpenGL and SDL also work well on all three major operating systems. Using Windows-specific solutions would do nothing but double the work.


I'm currently doing a .NET version, for no other reason that I need to learn the language so I can get a piece of paper so I can keep my job (that I don't really want to be doing anyway). Again, Visual Studio provide three types of application targets. Of course there are others.

harry666t wrote:Can you elaborate on sensors & tests? I think I don't fully understand the problem to which they are the supposed solution.

Sensors (or tests, or whatever) shouldn't make decisions on turning anything on or off. They should just send signals. "acts accordingly by turning on (start) or off (stop) effectors" - this sounds like a jump instruction that inspects its target memory address...


Yes, I was doing some research on assembly language. Refreshing to look at it after 13 years (did several classes on the Motorola 68HC11). And yes, I still haven't found a "graceful" way of determining what runs on the next cycle. What I do know is that based on the changes in the data values, effectors start and effectors stop. You didn't want a sensor to do work? They don't do work now. Just a set of flags to indicate that a condition changed. But something has to change the inqueue. For now I created a "sensor test" object to do exactly that. Louis's example is below:

-------------------
Comparison sensors--also called data sensors--must be associated with one or more effectors (see the discussion on sensor/effector associations below). Their function is to detect specific types of changes in their assigned data. As mentioned earlier, a signal-based system is a change-driven system.

As an example of how this works, let us say a given comparison sensor's job is to detect equality between two variables A and B. If A and B are equal both before and after an operation on either A or B, the sensor will do nothing. There has to be a change from not-equal to equal in order for the sensor to fire. The same change requirement applies to all sensors. COSA does not draw a process distinction between external (interrupts) and internal (changes in data) sensory phenomena.

-- From the Cosa Operating System, http://www.rebelscience.org/Cosas/System.htm#Sensors1
--------------------

harry666t wrote:Sensors can watch over data that sits in other components. Effectors also can act upon "foregin" data. If you run components separately, one by one, in any order, you effectively let sensors randomly run before or after effectors, thus creating one big race condition.


In Louis's world, data is encapsulated, or marked as private. And since there are no overarching sensors, sensors always run after the effector has changed the data.
-------------------------
High-level components do not have data at their level. They are just containers for other components. Most low-level components will have a data structure, the properties of which can be accessed by its cells. Components cannot directly access each other's data. That is to say, a component's internal data is always private. The only way a component may access another component's data is through an explicit request message. For example, a quick sort component may receive a request from another component to sort an array of integers. After completing the request, the sorter sends a 'service completed' signal back to the service requester.
The only data that may be shared directly between two components is message data. Even then, the destination component only has read permission. This is because two communicating components share a single message structure. The primary reason for the strict rule against data sharing is to prevent misuse. It would be too easy for one component to modify a property in another component without the knowledge of the other. This can potentially lead to incorrect assumptions and ultimately to catastrophic failure. In multi-threaded architectures, it is customary to use semaphores and other techniques to prevent two threads from simultaneously accessing a data structure. These techniques are not used in COSA because data sharing is forbidden.

From Software Composition in COSA, http://www.rebelscience.org/Cosas/components.htm
----------------------------
harry666t wrote:This ain't how open source works :)


And why did I put the PM thing for the source code? Because besides just the two of us, no one else is contributing anything. I'd like to have some auditing and getting feedback. Posting source code for the world in the per-alpha state that it is in probably isn't a good idea. As for a VCS, I'll look to see what is out there and let you know what I've decided. I use my own internal VCS so I know what I monkeyed with.

I'm revisiting Louis's design. For example, I'm thinking about attaching sensors to data values, so as an effector changes it, the data value tells the registered sensors to update their state immediately. I think a sensor needs to have a customized rule, rather than flags. Is the value at 100? Yes, then activate. Register flags are too generic. And the concept of components not visible to the processor is both scary and intriguing. Instead of the processor running components, the component's expose what effectors need to run and the processor runs those. That simplifies the number of running objects (a master inqueue/outqueue), but makes it more difficult to analyze if a logic flaw exists in a component.

It would be more encouraging if you had some positive feedback and ideas on what you would do on the things you don't like instead of "I just don't like it". I can't incorporate better ideas if they don't exist.
sparky
 
Posts: 22
Joined: Sat May 22, 2010 12:34 pm

Re: Creating a COSA virtual machine

Postby sprucely » Tue Sep 28, 2010 8:46 am

@sparky,

Correct me if I'm wrong but, from your description of tests, it sounds like you are assuming that if a sensor is in the buffer then it must fire a signal. If that is the case, I believe it is a wrong assumption because sensors contain the test logic and decide themselves whether or not to fire a signal. The concept of tests seems unnecessary to me.

Maybe you are trying to solve the same problem that is solved by effector-sensor associations as described at http://www.rebelscience.org/Cosas/System.htm? The way I handled these in my test cosa processor was to place all effectors in the buffer first and then add to the buffer with the associated sensors. That way all the changes caused by the effectors will be properly detected by all the associated sensors in the same clock cycle. A problem with my approach though is that it may be extra work to make it deterministic when scaled to multiple processors.

Also, there has been a source control repository for some time at http://bitbucket.org/sprucely/cosa/. I created a subfolder /UserProjects/ so people can maintain their own projects in it. If you set up a profile at bitbucket.org, I can add you to it.
sprucely
 
Posts: 24
Joined: Thu Mar 18, 2010 8:07 am

Re: Creating a COSA virtual machine

Postby harry666t » Thu Sep 30, 2010 2:13 pm

It would be more encouraging if you had some positive feedback and ideas on what you would do on the things you don't like instead of "I just don't like it". I can't incorporate better ideas if they don't exist.


Yes, and I feel a little bad about it. You do lots of work, but most of the time I just keep criticising it. I don't know, it seems to be my nature to be the contradictor (see older arguments in other topics on the forum, usually it's me vs. someone else). I've been born in the Chinese year of the Dragon, and in the rest-of-the-world's zodiac, I'm a Leo, so no wonder my ego's so high : |

Another thing is that I didn't want to suggest any ideas directly, like "no, just do it like this, see, it works in Reactor". You had an interesting idea of running each cell in a separate thread, and I've heard myths that Java is quite capable of running thousands of threads at once. If I told you "no, you must do it like this" we would have two almost identical implementations (and then why not just focus on one), so instead I thought of saying "no, this is wrong" when I perceived something as wrong.

I'm sorry if you feel discouraged because of me. I'll try to improve myself.

And why is that? In the complementary theory, something must be born if it is going to die.


Yes. But do you commit suicide each evening, in order to be revived, Jesus-style, in the morning? Or do you just go to sleep?

Are you suggesting that applications stay in memory in perpetuity?


Yes, they should stay in memory forever. Of course not in physical RAM, but ready to be swapped in & mapped into the address space at any moment.

Creating application instances is like going to forest, cutting a tree, making boards, assembling a door, and THEN closing it, whenever all you wanted is just to close a door. Killing application instances is like cutting that door into pieces and burning it whenever you wanted to open it. But because computers are superfast, and trees in computerland are free (they're just bits), people think it's OK. It's not.

The main problem with applications (and the sole reason why I hate the word "application") is lack of persistence. Applications crash randomly and I lose data. Applications are half-assed and do not persist some of their view state. Entire IPC APIs are shitty simply because they don't let me preserve my inter-application setup. To reuse the door analogy: OK, I got a new door, I didn't have to wait THAT long for it to close, but it doesn't have the poster of my favourite band on it anymore.

A rant combined with series of ideas and explanations follows.

You divided applications into three broad categories. I've got my own taxonomy, which is a little bit different - more "user centric" (after all, computers are nothing but OUR servants). I will argue that none of these categories demand that applications should ever be terminated, and what's more - NOT terminating applications actually makes them simpler and more usable.

--

* Applications provide services. An example service would periodically poll a POP account, fetch mail, and notify the user about any new messages. If the service is not needed at the time (something else is demanding memory), it should just be swapped out. If you don't need the service for some reason, there should be a "Don't Bother Me Now" button that pauses it.

I don't know how about you, but the only time I start my email program, is when I turn on the computer (which means: when it crashed, because I always use suspend to ram). Then it keeps running until the next restart. Prof. Andy Tanenbaum (of Minix fame) has his little mission: he wants no computer on Earth to have a "reset" button. I'm with him on this one.

And it's not just email program or IM program or any other regular desktop app that I want to keep sitting here and providing a service. If I want a small piece of code to listen for an event and do an action whenever the event arrives - I want it to do its job forever (or until I say "stop").

* Applications let you view and edit documents. Apps of this kind should act like seamless extensions to the desktop environment. The only thing they should ever do is to let you view and manipulate specific kinds of data.

How does this relate to persistence?

Let's take The Gimp (professional image editor) as an example. It takes ~12 seconds to start it on my machine (but once it's open, it needs only a split second to load an image). When I close it and open it again, it takes about 5 seconds to start. So we can assume that it takes about 7 seconds to read all the data from the disk, and the other 5 seconds is just some magic dance to initialise all the plugins.

So, in an "ideal world", The Gimp would start in 7 seconds when you first use it, and would start instantly if you used it recently. COSA has a potential to get us closer to that ideal, because there would be no plugins to poll at startup - only events like "plugin added", "plugin removed", "plugin updated". The image of the program would sit in memory, ready to open anything I need to open - instantly.

* Applications let you perform activities. Like playing a game, debugging a program, patching a synthesiser, organising a photo collection, designing a UI, etc. Activities are aggregations of services, viewers, editors, and other activities.

Taking the email app as our example. Communicating with people is an activity, which is performed through means of other activities - like writing or reading messages. Writing an email is another activity: a text editor is used, but there should also be links to other activities, like viewing the address book in search of the recipient's address. Reading messages involves listing, searching, filtering, displaying them - but also notifying the user of new unread messages.

Activities should aim to be technology-neutral. Email is only one channel of communication: there are also news, chat/IM, blogs, twitter, fora... If I wanted to catch up with what has been happening while I was offline, there should preferably be only one place to check it all: Activities -> Unread Messages (or something like this).

--

And here is the rant.

There are some more reasons why I think that killing app instances is stupid. I started doing music on my computer a while ago, and I've gathered a collection of various apps like soft synths, effect processors, sequencers, etc. I use JACK Audio Connection Kit to link them together. The problem is, I can't just tell the apps "OK, I'm done with playing right now, please come back exactly as I left you when I need you again, and now please stop wasting CPU and release the sound card". The reality: whenever I want to do some delightful synth sounds, I need to fire up QJackCtl, try to start the Jack server (while praying that Flash didn't claim /dev/dsp), fire up ZynAddSubFx, select a soundbank, fire up VMPK, then go to QJackCtl and connect Zyn with system audio output, connect VMPK Midi Out with Zyn Midi In, and only then I can start playing. Now try to do it with one hand while the other is playing some other, "real" instrument during a live, improvised performance.

There's LASH which enables JACK session persistence, but from my own experience, it just doesn't work. And there are lots of other settings I'd like to be persistent, which LASH doesn't cover. I want persistence baked right into the OS.

Uh, and one more thing. Hm, see, there's some random bug in the app. It divides by zero, tries to dereference a null pointer, or some other random stupid shit. What happens when the bug is triggered? It fucking crashes, losing all of my unsaved work (I *always* have unsaved work). Okay, some apps have automatic backup. But not all of them, and some really cool (but really unstable) effect processors are pissing me off because of that. See, I'm in the middle of a TOTALLY EPIC solo, and my guitar effect processor crashes. Or, my soft synth crashes: I don't even have a chance to save the work - I would need to save the setup every few seconds, because part of a performance with a modular synth is twiddling with the knobs, or sometimes even re-patching modules as you go.

Why did the application terminate? Couldn't it just look at the "last-good" state, and roll back to it? Or try to save the current setup and restart itself, as seamlessly as possible? Minix did exactly that - with fucking DEVICE DRIVERS, TWENTY YEARS AGO. Yeah, when you had a grave bug in a device driver, that would otherwise bring the whole system down - Minix would just restart the driver and continue operating like nothing ever happened.

End of rant.

That everything is loaded when the OS starts?


The OS doesn't need to be instantiated/killed either. We've had reliable hibernation&suspend across all major OSs for years.

Oh, and there was a really nice piece of a computer in the 80's, called the Lisp Machine. The main Lisp-M's feature (besides directly running Lisp bytecode on hardware) was that it was almost instant-on. The first "cold boot" took some time, but right after that it stored the current OS image in permanent storage. At any time you needed the Lisp-M, in order to become fully operational, it just needed to read the image into the memory.

For example, I have a sump pump that runs fairly regularly. It starts when the water level hits a certain point then stops when it has lowered the water to certain level. But if the power goes out, it doesn't turn on. This is my definition of termination. It's not going to start unless I get my generator going or the power company fixes their problem.


If I were to extend this analogy to applications - then I would understand it as "when I want to close Mine Sweeper, I turn off the computer by pulling the power plug from the socket".

To me, termination is analogous to physically destroying an object.

But the question is do you want a massive monolithic kernel or a microkernel?


A microkernel would be ideal from a "clean design" POV, but the only mainstream, OSS, MK-based OS seems to be Darwin. The main problem is driver support. See, driver support on Linux (which used to be a very "independent" project) sucked balls for a long time, but the pain was so big, the user demand so high, and there was so much work put into it, that at the moment the general case is that it's easier to get a random piece of hardware working on Linux than on Windows.

My dream is to create a self-hosting COSA-to-machine-code compiler and to use it to create the entire OS (from bootloader to desktop). I'll be taking a course on compilers this semester, so I guess I'm getting a little closer to this goal.

I'm currently doing a .NET version, for no other reason that I need to learn the language so I can get a piece of paper so I can keep my job (that I don't really want to be doing anyway).


: (

data is encapsulated, or marked as private. And since there are no overarching sensors, sensors always run after the effector has changed the data.


Data is private, but to a group of components, enclosed by a higher-level component. Read on.

High-level components do not have data at their level. They are just containers for other components. Most low-level components will have a data structure, the properties of which can be accessed by its cells. Components cannot directly access each other's data. That is to say, a component's internal data is always private.


This is old, outdated, and no longer true. The design has evolved (and will keep evolving in the future). Have a look at this post:

http://rebelscience.blogspot.com/2009/0 ... art-v.html

The communication method that I prefer is reactive data sensing in which components simply watch one another’s actions within a shared data environment. This way, sharers can immediately sense any change in relevant data and react to it if desired.


The most simple way to actually implement this is to permit data sharing between components at the VM level.

But yeah, Louis should update the original drafts, or at least provide links. These blog posts are very easy to overlook, while providing essential information.

Posting source code for the world in the per-alpha state that it is in probably isn't a good idea.


The "release early" part of the "release early, release often" rule. But okay, you can send me the current source, and I can have a look.

As for a VCS, I'll look to see what is out there and let you know what I've decided.


Come join us at BitBucket. Spruce can give you write access to his shared repo.

I'm thinking about attaching sensors to data values, so as an effector changes it, the data value tells the registered sensors to update their state immediately.


This is almost exactly what Reactor is doing, see here:

http://bitbucket.org/sprucely/cosa/src/ ... a.cs#cl-31

http://bitbucket.org/sprucely/cosa/src/ ... e.cs#cl-58

I wrote an overview of my VM in the readme:

http://bitbucket.org/sprucely/cosa/src/ ... .txt#cl-30

By the way, I rewrote Reactor in Python. It took a single afternoon. While C# is very powerful and pretty speedy, I need the flexibility of Python to have more room for fucking things up. Err, I mean, experimenting freely. Maybe I'll be porting stable features back to original Reactor, I guess it depends on PyReactor's performance (the main downside).

And the concept of components not visible to the processor is both scary and intriguing. Instead of the processor running components, the component's expose what effectors need to run and the processor runs those. That simplifies the number of running objects (a master inqueue/outqueue), but makes it more difficult to analyze if a logic flaw exists in a component.


Hm, you raised an interesting concern. The bytecode/cell graph would need something like debugging symbols. I've been thinking about possible memory layouts for the bytecode, and now I believe that some (maybe even full) information about every component should be embedded in VM's memory.

--

The way I handled these in my test cosa processor was to place all effectors in the buffer first and then add to the buffer with the associated sensors. That way all the changes caused by the effectors will be properly detected by all the associated sensors in the same clock cycle. A problem with my approach though is that it may be extra work to make it deterministic when scaled to multiple processors.


Hi Spruce! I've been discussing a similar issue with Louis on PM. I use sets (which pop their elements in non-deterministic order) for the buffers, so there was no way sensors could be added back to the "current" buffer without causing a race condition. My current strategy is to use a third buffer for sensors, and process that buffer in the usual way right after the main buffer is finished with. If you'd like to go multicore here, all you'd need is one more lock, so that all cores start processing sensors at once.

My main concern is that this approach can't really work on silicon. When the effector part of the die is running, the sensor part is idling, which means we can squeeze only about half the theoretical performance. So, here's the talk I've had with Louis on PM:

Actually, it's mainly a software issue. On a multicore processor, processing the censors of the previous cycle can happen while the next buffer is being processed. This can be done simultaneously, if there are enough cores or vector units. And come to think of it, there is no reason that the censors invoked during the previous cycle cannot be placed directly in the next buffer for processing. The only difference is that, when a censor fires, its output cells must be inserted into the current buffer for processing during the same cycle. This method can be used in software, too. However, there is no speed advantage in a software emulation.


OK, but. :D

The CPU/VM would have to carefully resolve all dependencies between sensors and effectors, as to avoid another race condition. I think that if this dependency resolution mechanism would have to be there anyway, it'd be better to process sensors in the "current" cycle and just put activated cells into the "next" buffer. It smells like a simpler solution.


You're right. I was also thinking that it may be OK to process the censors immediately as they occur. The only caveat is that the last invocation of a censor must be given priority and must be able to revoke the previous one if necessary. Revocations should be rare, in my opinion.


I believe it is still a long time before we get to implement COSA on silicon, so this seems to be somewhat irrelevant right now.
harry666t
 
Posts: 48
Joined: Thu Mar 11, 2010 7:38 am

Re: Creating a COSA virtual machine

Postby sparky » Thu Dec 09, 2010 2:06 pm

Unlike my previous posts, I'll keep this brief. I decided to revisit the articles written by Louis and see 1) was I missing something and 2) organize his thoughts into functional pieces. The result is the attachments below.

I see the reasoning and envision how to implement it exactly as described. If I had more free time, I'd sketch out a technical design, which I would do first before continuing on. I would simplify the platform: use a browser as a client and have a back-end server update the browser. The component library reminds me of the UDDI work that has been done.

This approach does have it's drawbacks: I/O limited to what the browser can provide. But the number of variables is simpler.

Happy holidays.
Attachments
COSA Requirement Specification.zip
Requirements based on article text
(31.02 KiB) Downloaded 679 times
COSA Primer.zip
Article text with indicated sentences
(93.74 KiB) Downloaded 668 times
sparky
 
Posts: 22
Joined: Sat May 22, 2010 12:34 pm

Previous

Return to COSA Virtual Machine

Who is online

Users browsing this forum: No registered users and 1 guest

cron