Luca Bolognese's Blog

The Hooks

Tue, 05 May 2026 00:00:00 GMT

The money was gone. The husband’s business had dried up.

Elena had long black hair. She was small. Her hands were soft. She had never used them for work.

She went to the village and to the butcher.

Bruno was a big man with thick arms. His hands were red and split at the knuckles.

“You cut?” Bruno said.

“No.”

“You will,” he said.

“I will learn.”

“Apron.”

The shop smelled of sawdust and meat.

“Not like that,” Bruno said.

She did not stop.

“Again.”

She did it again. Bruno took the parcel and pressed it flat with his palm.

“Too loose.”

She did not change it.

Bruno looked at it. “Again.”

She tightened the string.

“Better,” he said.

The next day, a man came in. Tall and thin.

“She doesn’t talk.”, he said.

“She works,” Bruno said.

The man looked past him. Bruno did not move. The man paid and left.

One day, rain fell hard against the glass. The shop was empty.

“Lock the door,” Bruno said.

She locked it.

Bruno finished the cut and set the knife down. He wiped his hands.

“Come,” he said.

He took her wrist.

They went into the cellar.

The hooks hung in their rows, pale and still.

After, Bruno set the knives in the sink.

“Wash.”

She washed them. The water ran pink, then clear.

At dinner Tomas watched her hands.

“You work late,” he said.

“Yes.”

He cut his food.

“You stay late.”

She did not answer.

He nodded.

The next day, the door of the shop was unlocked.

Tomas entered. He saw them go into the cellar. He did not move.

When they came out, he stepped forward.

“You forgot to close the door,” he said.

Bruno looked at him. “No.”

Tomas nodded.

Later, it was warm and they all sat on the terrace.

“The meat is good,” Tomas said.

“It is the best,” Bruno said.

Tomas poured the wine.

“You should come to the sea with us,” Tomas said.

Bruno looked at him.

Tomas waited.

Elena did not move.

Tomas saw her and did not look away.

“Yes,” Bruno said.

Inside the house, Tomas stood.

“Come,” he said.

They all went into the bedroom.

The door closed.

Later, in the shop, the air was cold. The hooks hung in their rows, clean and still.

Elena stood with the paper in her hands. Bruno was at the block. Tomas stood in the doorway.

She did not look away.

A hook moved once. Slow.

Too Light to Hold

Thu, 16 Apr 2026 00:00:00 GMT

Dust lay thick on the path and held her weight. The sun was low. It was a long walk to the river, but the house was hot and smelled of tobacco and the men. Maria walked with her hips stiff. Red marks showed at the small of her back. Her thighs carried a dull, heavy heat.

The river ran fast. Cold. It broke over grey stones in the shallows and made a sound like wind moving through leaves. Maria sat on a flat rock and took off her boots. She stepped into the water. Skin-stinging cold. Good. The current pulled at her ankles, trying to carry her downstream, but she stood firm. The river did not care about the house or the money. It moved.

She took a piece of grey lye soap from her pocket and began to wash. Her legs first. Then the places where the men had been. Cold water over her skin. Quick. Hard. A kingfisher dived into the pool near the reeds. It struck the water fast and came up with nothing, then lifted to a high branch to wait again. Maria watched it and scrubbed until her skin was pink and the soap was gone.

She sat on the bank to dry. A yellow butterfly landed on a piece of damp, rotting drift log near her feet. It opened and closed its wings. Bright against the rot. It rested for a moment, searching in the decay, and then a gust of wind took it. Too light to hold, carried toward the dark trees and gone.

Evening chill. She pulled on her dress. The fabric was damp and clung to her skin like a second layer of sweat, but it was clean. The working face coming back. The mask settling into place. She looked at the river one last time. Darker now. It carried the silt and the suds and the memories of the afternoon away toward the sea. The river stayed clean by moving. She could not stop.

Back up the path. The incline shortened her breath. The house appeared through the trees, its porch light a yellow smear against the purple sky. Moths gathered thick around the bulb, striking the glass with soft, blind thuds.

The men were there. In a line near the fence. Leaning against the wood or checking their watches. The Baker, smelling of flour and stale sweat. The Clerk with the nervous eyes who always looked at his boots.

And the Young Man.

A clean white shirt. Hair combed back with water. He saw Maria coming up the path and stepped forward. His face held an imagined thing. He looked at her as if they shared a secret. As if he were the one who would take her away from the river and the house and the dust. He stood as if his love would hold.

Maria looked at him. The white shirt. The hope in his eyes. The yellow butterfly on the rotting log.

No malice.
No hate.
Only the weight of the night beginning again.

She nodded as she passed the line.
He smiled.
A quick, breaking smile.
He thought it would hold.

It only meant he was next.

I wrote a C runtime from scratch

Fri, 27 Mar 2026 00:00:00 GMT

I wanted to understand what happens between my C code and the hardware. Not in theory — I’ve read the textbooks. I wanted to feel it: write the syscalls, handle the startup, own every byte in the binary.

So I wrote a small C runtime from scratch: lucabol/laststanding.

It’s three header files. l_os.h gives you the basics — strings, memory, file I/O, processes, pipes, terminal control. All implemented with direct syscall wrappers, no libc anywhere. l_gfx.h adds pixel graphics — drawing primitives, a bitmap font, keyboard and mouse input. l_ui.h builds on that with immediate-mode UI widgets: buttons, sliders, checkboxes, text inputs.

It runs on Linux (x86_64, ARM, AArch64) and Windows. Binaries are statically linked, stripped, and typically 2–10 KB. A hello world on ARM compiles to 676 bytes.

The repo includes a dozen small Unix-style utilities (grep, sort, wc, ls), a Vim-style text editor, a shell that can run itself, a snake game, a Mandelbrot renderer, and several graphical demos — plasma, starfield, Doom-style fire, Conway’s Life, an analog clock. Plus a couple of UI demos showing forms and sliders. Everything compiles with -ffreestanding -nostdlib and nothing links to libc.

The whole thing is built around static inline functions, so you only pay for what you use. On Linux, syscalls go through inline assembly. On Windows, it calls Win32 APIs directly with UTF-8 to UTF-16 conversion at the boundary. The graphics layer writes to the Linux framebuffer or opens a native GDI window. The UI is immediate-mode: no heap, no widget tree, just declare your widgets every frame.

AI helped enormously. I used it to scaffold the initial code, write tests across four architectures, debug platform differences, and iterate on the API design. What would have taken weeks of manual cross-platform plumbing happened in days. The CI runs 22 targets — Windows, Linux, ARM32, AArch64, each with gcc and clang — and AI helped get them all green.

It’s not a production library. It’s a learning project that got slightly out of hand. But there’s something satisfying about a binary that contains nothing you didn’t put there yourself.

Notes: The Metamorphosis — Franz Kafka

Fri, 27 Mar 2026 00:00:00 GMT

Part I: The Discovery

The Transformation: Gregor Samsa, a traveling salesman, wakes up to find himself transformed into a “monstrous vermin.”
Anxiety Over Work: Despite his condition, his primary fear is being late for his train. He is the sole financial provider for his parents and sister.
The Reveal: His office manager arrives to check on him. When Gregor finally opens the door, the manager flees in horror, and his mother collapses.
The First Injury: His father violently herds Gregor back into his room using a cane and a newspaper, causing Gregor to bleed as he is shoved through the narrow doorway.

Part II: Adaptation and Isolation

Grete’s Care: His sister, Grete, becomes his sole caretaker. She discovers he now prefers eating rotting scraps over fresh food.
Losing Humanity: Grete and their mother decide to remove the furniture from Gregor’s room to give him more space to crawl. Gregor feels his humanity slipping away and tries to “save” a framed picture of a woman in furs.
The Apple Incident: Seeing Gregor out of his room, his mother faints. His father, misinterpreting the situation, chases Gregor and pelts him with apples. One apple lodges in Gregor’s back and is left there to rot, causing a severe, permanent wound.

Part III: The Decline and Aftermath

Financial Strain: The family’s finances dwindle. They take in three boarders to make money, and Gregor’s room becomes a storage area for trash and old furniture.
The Violin Performance: Drawn by the sound of Grete playing the violin for the boarders, a dusty and weakened Gregor creeps into the living room.
The Breaking Point: The boarders are horrified and refuse to pay. Grete, exhausted by the burden, declares that the family must get rid of “it,” claiming that if it were truly Gregor, he would have left them long ago to spare them suffering.
Gregor’s Death: Hearing this, Gregor retreats to his room. He dies that night, feeling a sense of love and “empty peace” toward his family.
Renewal: The family feels an immediate sense of relief. They take a day off, go for a walk in the countryside, and realize that Grete has grown into a beautiful woman with a bright future ahead of her.

Notes: The Picture of Dorian Gray - Oscar Wilde

Fri, 27 Mar 2026 00:00:00 GMT

Part I: The Innocent Muse

The Portrait: The talented artist Basil Hallward paints a full-length portrait of the strikingly beautiful and innocent young Dorian Gray.
Lord Henry’s Influence: At Basil’s studio, Dorian meets Lord Henry Wotton, a cynical aristocrat who espouses a philosophy of “New Hedonism,” claiming that youth and beauty are the only things worth living for.
The Fatal Wish: Terrified of aging, Dorian utters a fateful wish: that he could remain young while the portrait bears the marks of time and his sins instead.
Sybil Vane: Dorian falls in love with a young, talented actress named Sybil Vane. However, after he brings Basil and Henry to see her perform, she acts poorly because her love for Dorian has made her realize the “hollowness” of the stage.
The First Change: Dorian cruelly rejects Sybil, telling her she has killed his love. When he returns home, he notices a touch of cruelty in the mouth of his portrait. Sybil commits suicide shortly after.

Part II: The Life of Sin

The Hidden Room: Following Lord Henry’s advice to view Sybil’s death as a “theatrical tragedy,” Dorian embraces a life of sensory pleasure. He hides the portrait in a locked attic to conceal its physical decay.
The Yellow Book: Lord Henry gives Dorian a “poisonous” French novel that chronicles the life of a man seeking every possible sensation. This book becomes Dorian’s blueprint for the next 18 years.
The Social Ruin: As the years pass, Dorian remains physically flawless, but rumors circulate about his “corrupting” influence. Many of his friends are socially ruined or driven to suicide after associating with him.
The Murder of Basil: Basil Hallward confronts Dorian about these rumors. Dorian shows Basil the hideous, rotting figure in the portrait. When Basil begs him to repent, Dorian, overcome by a sudden burst of hatred, stabs Basil to death.
Disposing of the Evidence: Dorian blackmails an old friend, a chemist named Alan Campbell, into using chemicals to dissolve Basil’s body.

Part III: The Final Reckoning

James Vane’s Revenge: Sybil’s brother, James, hunts Dorian down to avenge her death. He is nearly successful but is momentarily fooled by Dorian’s youthful face. James is later killed accidentally during a hunting party.
A Failed Attempt at Virtue: Dorian decides to “be good” and spares a young girl’s heart to see if the portrait improves. Upon checking the painting, he sees that the face has become even more hypocritical and loathsome.
The End of the Mirror: In a fit of rage and desperation to destroy the evidence of his soul’s decay, Dorian stabs the portrait with the same knife he used to kill Basil.
The Transformation: A crash and a scream are heard. The servants enter to find the portrait restored to its original, beautiful state. On the floor lies a withered, wrinkled, and “loathsome” old man with a knife in his heart, identifiable only by his rings.

I built my own notes app (with a lot of help from AI)

Thu, 26 Mar 2026 00:00:00 GMT

I recently moved away from a paid notes product and built my own replacement: lucabol/notes.

For a long time I used a paid notes product because it was convenient, polished, and already there. But over time I realized I was mostly paying to work around someone else’s idea of how my notes should behave.

So I did the most reasonable thing possible: I replaced it with a custom-made markdown notes tool built surprisingly fast with AI.

Now my notes are just plain files. I have a CLI, a desktop GUI, import tools, tags, search, and no monthly fee quietly judging me from my credit card statement.

What I like most is not just that it works — it feels mine. It matches how I think, how I organize things, and how I want the tool to evolve.

Also, there is something deeply satisfying about leaving a subscription behind and replacing it with a folder full of markdown and a slightly unreasonable amount of PowerShell.

AI made the build dramatically faster, but the interesting part wasn’t speed alone. It let me iterate on ideas quickly, fix annoyances immediately, and turn “I wish this app did X” into “fine, now it does.”

It’s not a giant startup product. It’s just my notes app. And honestly, that’s the whole point.

A Good Life

Thu, 12 Mar 2026 00:00:00 GMT

She woke late to a throbbing headache and the rhythmic flash of her phone’s blue light. Five likes and one dislike. One dislike! Her morning was ruined. She wondered if it was the new hair, but then dismissed it. It was the latest fashion; everyone was doing it. That couldn’t be it.

She looked in the mirror. Her chubby face stared back, but she wasn’t worried; a chubby face was an easy fix. Her stream filter would streamline her jaw and deepen her skin tone, an “earth-toned” aesthetic was in vogue this week. Her natural skin was too pale for the Top 5. She knew the metrics; hence, the filter.

The team had sourced a boyfriend for the week, selecting him from a pool of hundreds. They were streaming together later. In post-production, she would deepen her voice slightly to sound more “trustworthy.” The sound guy would handle the frequencies.

Her morning routine was a work of art. It had been voted “BEST MORNING ROUTINE” in the 12-15 category two years running. The camera was hidden behind the bathroom mirror, capturing her as she brushed her teeth. Sometimes she would sing in the shower, not too well, not too poorly. You don’t want to show off. People hate that.

She cultivated her mediocrity with intense discipline. It was her secret weapon. She wasn’t too funny, just enough to prompt an easy smile. She avoided anything difficult or deep. You don’t want to make anyone think.

Her day was planned to the minute. You can’t make things up as you go; that’s the fastest way to drop in the CHART. You have to live deliberately.

The morning was for “activities,” playing with the dog, for instance. Her ratings always spiked when she did that. But these things are temporary; the audience will eventually tire of the dog. She’ll have to phase him out. Maybe a “tragic accident” or a disappearance. Tragedies are excellent for the algorithm. For now, though, she cuddled him like there was no tomorrow. She called him Fluffy, the top choice of the focus group. Every detail counts.

At lunch, she would perform the weight-loss arc. The standard script: the struggle, the relapse, the “journey.” Maybe some purging? Definitely some tears. She had the eye drops ready in her kit.

The afternoon was equally full. That was when the “relationship” story would unfold. Maybe a breakup today, maybe not. It all depended on the live audience reaction. There were plenty of other accessories, other boyfriends, waiting in the wings.

The evenings were for the chat. That was the trickiest part: staying lightly informal. She had to make it look personal without ever saying anything actually personal. Above all, she stayed away from the substantive. If you say something controversial, someone is bound to disagree. Disagreement is dangerous. It is not rocket science, folks.

Before bed, she checked the Chart one last time. She had climbed a dozen positions in the past year alone. Not first class yet, but respectable.

As she closed her eyes, she felt a deep sense of serenity. This was the life she had worked so hard for. She might as well enjoy it.

When the Tide Comes

Sat, 03 Jan 2026 00:00:00 GMT

The old man moves slowly now. It takes him longer to walk all the way to the beach. He goes through the mangrove trees following the narrow path to the shore. The sky is clouded. The wind is strong and the trees sway.

When he reaches the beach, he stops. He takes off his sandals and walks on the wet sand. It is winter and he knows he should not do that.

He looks at the road through the trees. The woman from the village arrives in an hour on her bicycle. She is forty now, he thinks. He never asked her age. They do not talk much. She comes to clean the house but that is not it.

He suggested it. Once a week, he pays her well. The husband is a fisherman and is away. He doesn’t mind the cash, the old man reckons.

They drink wine first. It gives her strength. There is nothing to like in an old man. But she comes and does what he asks. No passion, just the act. That is enough.

Afterward she makes coffee. They chat. She tells him about her kids and he gives advice. She listens. Sometimes she smiles.

The old man watches the sea. The waves are rough. The wind is cold. He does not know about kids. He was never a father to his child. He was on the oil rigs, earning a living, living away from them.

Now it is late. His son is in another country. Finance or marketing. They talk on the phone. He does not understand the boy. The boy has a child of his own and they never visit.

A gull flies over him, crying. The old man hunches to avoid it. His back hurts. He stands up and continues walking.

It is a small beach. A stretch of sand between the trees and the sea. When he was home from the rigs, he came here with his wife. They made small talk and then there was silence. She resented him for being away. He didn’t blame her.

Now he misses her. He misses the silence.

There is a rock by the water. The dog used to run around it chasing the birds. A big white dog. He would be behind the door when he came home and wag his tail.

The old man reaches the end of the beach. An old house sits on the rocks. The couple moved away years ago. They used to have cocktails on the porch. The man of the house liked the old man’s wife too much. He drank too much. It does not matter now. It never mattered.

The old man turns back. The wind hits his face and he staggers. He closes his eyes and breathes. He feels the cold in his chest. It is a sharp cold.

He sits down by the rock. The sand is wet and the tide is coming in. He thinks about the white dog. He does not see the dog, but he remembers the weight of the dog’s head on his knee.

He lies back against the stone. The wind blows the spray from the waves over him. He does not get up.

They find him a week later. The birds did not touch him. The tide had come and gone, leaving salt on his coat.

The Girl of the Lake

Wed, 29 Oct 2025 00:00:00 GMT

Twice in my life, I saw the girl’s image in the lake. The first time I was just a little girl. My mama had sent me there to wash the dishes. I was walking down the pier when I saw it.

They say the lake remembers. They say the lake is an old man with blue eyes who sleeps and dreams about us. They say a lot of stuff about the lake, but I never believed any of it. I was a child and didn’t believe in magic.

The image I saw was just a reflection, like many the lake throws your way now and then to confuse you. A speck of light in the corner of your eye, quickly forgotten.

But this was different. The image stayed with me. It stayed with me all these years. I didn’t understand it then. Now I do.

I saw the image again yesterday. It was clearer this time. The old man had decided that I was worthy of its confidence. I was ready to see.

It was a girl. She looked sad, as if something bad had happened to her, as if asking for help. Her eyes open wide, like a scared deer.

I know the girl of the lake. We grew up together. We would build boats with the leaves of the mangrove trees and laugh loudly when the lake pushed them around. The old man - the lake - would drown our boats, but we didn’t care. We would build new ones.

Later on, we went dancing together. We drank together and did foolish things. She kissed the sawmill boy—crooked teeth, broad shoulders, red hair. The kind that never stays.

The girl was ashamed of it because he was no good and because he left the day after and because there was no future in it. I told her that no one ever died because of a bad kiss, but it wasn’t just a kiss. It was worse. It had consequences.

Her father beat her badly because of it. I don’t know how he found out. Maybe someone saw them. She walked with a limp and her eyes were puffy and she stopped laughing.

The girl married the town barber. He didn’t mind her big belly, he said. He was crazy for her, he said.

Then the child came and it all went to hell. Love turns bad when a child cries all night and you don’t have any money and people laugh at a son that is not your son.

The girl went away. Took the child to the big city far away and forgot, but she didn’t forget. She lived and she laughed and drank and danced and was often ashamed because of it all.

Her hair went gray and her son grew up and she learned to live with it because she found the beauty in her sadness.

As I was walking by the pier, a little limp in my step still, the wind picked up. A wave danced over the old man. It made tiny ripples in the water.

The image was broken briefly, but then came back, clearer than before.

And there it was. My own reflection.

The lake remembered. The old man dreamed of me. I understood then.

The Death of Henry

Tue, 28 Oct 2025 00:00:00 GMT

It was about to happen. Henry knew it. The process was complete. The moment had arrived. He was going to die.

The external world held no interest for him anymore—just one disappointment after another. Grey nights followed grey days. Some happiness, some sadness. Nothing to write home about, really.

Henry had taken the best life had to offer. He had money. He had women. He had friends. He traveled the world. He saw everything. But none of it meant anything. Nothing made sense.

Death, though—death was different. Death was interesting.

Henry had been working on it for a while, willing himself to die. Day after day, night after night, he would lie in bed, trying to lower his heartbeat, relax his muscles, slow his breath. But it isn’t easy to will yourself dead. It takes control. A lot of it. It is the ultimate test.

Today, he was sure, was the day. He was ready. Or so he hoped…

But it was not to be. Every time he got close, something pulled him back. His mind refused to let go. But why? What was holding him back?

He had no friends. No family. No interests. So why then? What was he living for?

He wondered. And the night came. And he dreamt of her.

She was dancing on the pier, like she often did. The moon shone over them both. The sea sang. The wind screamed. He walked toward her, but she floated away in the wind. He was alone again, in the dark.

He never understood the dream. He didn’t know the girl. He had never met her.

There were prophecies, of course. He knew that. There are always prophecies. He had seen the signs—the red star, the voices, the lights.

Maybe it was the girl keeping him alive. Maybe she meant something…

And the months passed. And the years passed. And still, he did not die. Everyone around him was gone. He was old—very old—but he remained.

Then he saw her.

She was walking her dog one warm summer evening. He tried to speak to her, but she ran away, frightened by his appearance. But he didn’t give up. There was too much at stake.

He found her again. He made her listen. He had to restrain her. He talked and talked, deep into the night. At times, he had to be forceful. Why wouldn’t she just listen?

This went on for days. Weeks. Maybe months. It was hard to tell. He lost track. Eventually, she stopped answering his questions.

The day they found him, it was hot. He was sleeping on the floor, surrounded by rotten food and strange books. Scary books, with devils and angels. Tarot cards. Saints.

A skeleton was chained to the radiator. It made the front page of the local newspaper.

I was given Henry’s books. I’ve read them all. Twice. Some pages are blank. Others seem to shift when I look away.

I’ve started keeping notes. I’ve seen the red star. I’ve heard the voices.

I think I saw her once—walking her dog, just like he said.

I’m not sleeping much anymore.

Need to find her.

Winter feelings

Wed, 17 Sep 2025 00:00:00 GMT

Tom Campbell goes to the desk to punch his card.

“Good session, professor. Are you training for something?”, the woman says, beamily.

He flushes.

“If you want to take your training to the next level, I am a personal trainer.”

Tom looks at her. “I am fine.”

“Okay, see you next week,” eyes already on the next customer.

Tom walks to the bookstore, picks up a book, reads the back cover, puts it back. Picks up another book, same.

He walks to the coffee shop, orders a coffee, sits down by the window. Looks at the empty chair across from him. Sighs.

He takes out his phone. A picture of his wife. Smiles. Puts the phone away. Outside it is snowing.

“Sorry to bother you, but I saw you in the bookstore. I want to learn about investing.”

Tom looks up at the woman. She is young, with an accent. His eyes wander to her legs, to her arms. He looks away.

“I used to teach.”, he says.

He gives her a few book recommendations. She thanks him and leaves. Tom watches her go. He picks up his coffee and takes a sip.

He goes home. The doormat is crooked. He fixes it. Hangs up his coat, lines up his shoes. The apartment is quiet.

He touches the photo of his wife. Takes it down. Pauses. Puts it back.

In the kitchen, he makes a sandwich and eats it standing.

He watches sports, then a documentary, then the news. Picks up a book. Puts it down. Looks at the photo again.

Outside, snow falls.

A week later, she is in the queue to get coffee. She looks at him. He moves his hand to wave, but she is staring past him. He turns his chair toward the window. She didn’t see him.

“Tom?”, a voice from behind.

She smiles. “We met last week in the bookstore. You helped me with some books.”

“Oh, yes, of course. I’m sorry, I have a bad memory for faces.”, he lies.

She sits down and they talk. Tom waves his hand around, animatedly. She listens with wide eyes, interjecting questions. Tom’s eyes wander to her legs. He clears his throat and looks away.

“Can I have your phone number? In case I have more questions.”, she asks.

He writes his number on a napkin, his hand shakes. She smiles and leaves with her coffee. Tom watches her go from the window. He sips his coffee. It is cold now.

A few days later he gets a message from her. The question is very difficult.

“It is too long for a message”, he writes back.

“Maybe we could meet for coffee?”, she replies.

They meet again. She sits closer to him and waves her hands around. Tom nods approvingly.

“My wife was not very interested.”, Tom says, hesitating. He cleans his glasses nervously.

She looks at him, waiting.

Tom stays silent for a long time.

“I am sorry.”, he stammers.

She puts her hand on his. “It’s okay.”, she says.

They talk for a while.

“Would you like to meet again?”, she asks, “I will call you.”

They go to dinner. She talks.

Her body sways. Tom looks at her lips, her eyes, her hands.

She touches his hand. He shakes. She looks at him, smiles and says nothing.

When the coffee arrives, their chairs are closer together. Tom’s glasses are fogged up. He cleans them, looks at her.

“My car is over there. I can give you a ride home.”, she says.

“Maybe we could …”, he starts.

“I would like that.”, she replies, smiling.

They go to his apartment. She looks around. They sit down on the couch. Tom takes her hand and she doesn’t pull away. They kiss.

He is shaking, but she is gentle. She guides him, shows him what she likes. They make love and fall asleep in each other’s arms.

The morning comes and she is gone. There is a message on his phone.

“Thank you for last night. I will see you soon.”

But she doesn’t call or message.

Tom cleans the apartment, goes to the gym, bookstore, coffee shop. He waits for her message. Days pass. He can’t sleep. He tries to call her, but the phone rings and rings. No answer.

He sends a message, but no reply.

Then: “Crazy week. Can we meet?”

They meet at the café. She wears a new coat. Her nails are done. She kisses his cheek, sits down.

“There’s a seminar in Geneva,” she says. “It’s expensive. But it could change everything for me.”

Her hand rests on his arm.

“I trust you.”

He nods and stirs his coffee.

“You’re amazing,” she says, smiling.

They walk together. She laughs at things he doesn’t say.

At his house, she looks around. Touches the books and the photo on the wall.

She takes off her coat. Sits on the couch. Crosses her legs and looks at him.

He sits beside her. She leans in and they kiss. Her hand moves to his chest. His hand to her waist. They move to the bedroom.

In the morning, she is gone.

A note on the table: “Thank you. I’ll call you.”

He waits. Days pass. No call. He sends a message. No reply.

He sees her across the street, from the coffee shop. She is with another man. Younger. She wears the same coat. Her hand is on his arm and she laughs. The man smiles.

Tom watches, but she doesn’t see him. They get into a taxi and drive away.

Tom rises and walks home.

The doormat is crooked. He fixes it. Takes off his coat. Puts away his shoes.

He looks at the photo on the wall and straightens it. Goes to the kitchen and makes tea.

He sits by the window. Snow is falling and he watches it for a long time.

Then he stands up and smiles.

Notes: The Iliad - By Homer

Tue, 09 Sep 2025 00:00:00 GMT

The Iliad: A Chapter-by-Chapter Summary

Book 1: The Rage of Achilles

The Achaean (Greek) army has been fighting the Trojans for ten years.
The priest Chryses asks the Greek commander, Agamemnon, to return his captured daughter. Agamemnon’s refusal leads to a plague sent by the god Apollo.
To end the plague, Agamemnon agrees to return Chryseis but takes Briseis, the war prize of the great warrior Achilles, as a replacement.
Enraged by the insult, Achilles withdraws from battle. He prays to his mother, the sea-nymph Thetis, to ask Zeus to turn the war against the Achaeans.

Book 2: The Dream and the Catalogue of Ships

Zeus sends a deceptive dream to Agamemnon, telling him he can take Troy.
Agamemnon tests his troops, and they try to flee, but Odysseus rallies them.
The book provides a detailed catalogue of ships, listing the Greek contingents and leaders.

Book 3: The Duel of Menelaus and Paris

The Trojan prince Paris challenges the Achaean king Menelaus to a one-on-one duel to decide the war.
Before the duel, Helen points out the Greek heroes from the walls of Troy to King Priam.
Menelaus dominates Paris, but before he can kill him, the goddess Aphrodite saves Paris.

Book 4: The Truce is Broken

The gods debate the fate of Troy. Athena incites the Trojan archer Pandarus to shoot an arrow at Menelaus, breaking the truce.
A full-scale battle erupts.

Book 5: Diomedes’ Aristeia

The Achaean hero Diomedes, empowered by Athena, goes on a rampage.
He wounds the gods Aphrodite and Ares in a great display of valor known as an aristeia.

Book 6: Hector and Andromache

The Trojan hero Hector returns to the city to find his mother and wife, Andromache.
He shares a tender farewell with his wife and infant son, Astyanax, before returning to battle.

Book 7: The Duel of Hector and Ajax

Hector challenges any Achaean to a duel.
The great warrior Ajax (the Greater) accepts, but the duel is stopped as night falls.
Both sides agree to a temporary truce to bury their dead.

Book 8: The Achaean Rout

Zeus forbids the gods from interfering and tips the scales of battle in favor of the Trojans, driving the Achaeans back to their ships.

Book 9: The Embassy to Achilles

The Achaeans are in despair. Agamemnon sends a delegation, including Odysseus, Ajax, and Phoenix, to Achilles.
They offer him many gifts and the return of Briseis if he will rejoin the fight.
Achilles, still filled with pride and rage, rejects the offer.

Book 10: The Doloneia

Odysseus and Diomedes sneak into the Trojan camp at night.
They capture and interrogate the Trojan spy Dolon before killing him and slaughtering a number of Thracian soldiers.

Book 11: Agamemnon’s Aristeia

Agamemnon performs a great deed on the battlefield, but he and other key Achaean leaders are wounded.
The Trojans continue to push the Achaeans back.

Book 12: The Trojans Storm the Wall

The Trojans, led by Hector, successfully break through the Achaean defenses and storm their camp.

Book 13: Poseidon Rallies the Achaeans

The god Poseidon secretly intervenes, bolstering the morale of the Achaeans and turning the tide of battle.

Book 14: The Deception of Zeus

Hera seduces Zeus and lulls him to sleep, allowing Poseidon to further aid the Achaeans without Zeus’s interference.

Book 15: The Achaeans Driven Back to the Ships

Zeus awakens and is furious. He forces the other gods to stop meddling and orders Apollo to give Hector new strength.
Hector drives the Achaeans all the way back to their ships.

Book 16: The Death of Patroclus

Achilles’ best friend, Patroclus, wears Achilles’ armor to inspire the Achaean troops.
Patroclus disobeys Achilles’ warning and is killed in battle by Hector, with the aid of Apollo.

Book 17: The Fight over Patroclus’ Body

A fierce battle rages over Patroclus’s body and armor, as Menelaus and Ajax fight to recover it.

Book 18: The Shield of Achilles

Achilles learns of Patroclus’s death and is overwhelmed with grief and rage.
He decides to re-enter the battle. His mother, Thetis, has the god Hephaestus forge him new armor, including a magnificent shield of Achilles.

Book 19: Achilles’ Reconciliation

Achilles receives his new armor and reconciles with Agamemnon, who returns Briseis and offers many gifts.
Achilles officially re-enters the battle.

Book 20: The Gods Join the Battle

Zeus allows the gods to directly intervene, leading to chaotic duels.
Achilles’ rage is fully unleashed on the Trojans.

Book 21: Achilles’ Rampage and the River Scamander

Achilles slaughters many Trojans.
The river god Scamander becomes angry at the number of bodies and rises against Achilles, who is only saved by the intervention of the gods.

Book 22: The Death of Hector

Achilles chases Hector around the walls of Troy.
Athena, disguised as Hector’s brother, tricks him into stopping and facing Achilles, who kills him and then dishonors the body.

Book 23: The Funeral Games for Patroclus

Achilles holds elaborate funeral games in honor of Patroclus, including chariot racing, boxing, and wrestling.

Book 24: Priam and Achilles

King Priam, guided by the god Hermes, journeys to the Achaean camp to beg Achilles for Hector’s body.
Achilles is moved to compassion and returns the body, granting a truce for the funeral rites.
The poem ends with the burial of Hector.

History of the text and canonical sources

The canonical text of the Iliad doesn’t come from a single, original historical document, but rather from a long tradition of oral and written transmission. The oldest and most important documents that inform our modern version are papyri and medieval manuscripts.

Papyri

The earliest surviving fragments of the Iliad are on papyrus scrolls. These are crucial because they date back to antiquity, providing a direct link to the text as it existed in the Greco-Roman world. * The oldest fragment: The earliest known fragment is the “Derveni Papyrus” from the 4th century BC, though it contains commentary on a section rather than a direct copy of the text. * The most significant papyri: Many papyri fragments, some dating to the 3rd century BC, have been found in places like Oxyrhynchus, Egypt. These often contain only parts of the poem, but they show that the text was relatively stable, although small variations and different versions did exist.

Medieval Manuscripts

Medieval manuscripts are complete copies of the Iliad text, written on parchment or paper. They are the most complete and valuable sources for the canonical version of the poem. * Venetus A: This 10th-century manuscript is considered the most important source for the Iliad. It contains not only the full text of the poem but also extensive scholia—ancient marginal notes and commentaries by scholars like Aristarchus of Samothrace and Aristophanes of Byzantium. These notes often discuss different versions of lines, textual variants, and interpretations, making the manuscript an invaluable resource for understanding the history of the text. * Venetus B: Also from the 10th century, this manuscript is another key source. It contains similar scholia and helps to corroborate or clarify the readings in Venetus A. * Other medieval manuscripts: Many other Byzantine manuscripts exist, which scholars use to compare readings and create a comprehensive picture of the text.

In short, the canonical Iliad is a reconstructed text based on a critical analysis of these various documents. Modern editions are created by comparing the readings from the fragmentary papyri with the more complete medieval manuscripts to arrive at a version that is believed to be the closest to the one used in the ancient world.

Notes: A general intro to the Great Books - By Adler

Tue, 09 Sep 2025 00:00:00 GMT

1. Plato, Apology and Crito

Apology
- Socrates defends himself against charges of corrupting the youth and impiety.
- He recounts the Oracle at Delphi’s claim that no man is wiser than he, which led him on a quest to question others.
- He argues that his mission is divine and that an unexamined life is not worth living.
- He is found guilty and sentenced to death.
Crito
- Crito visits Socrates in prison with a plan to help him escape.
- Socrates rejects the offer, arguing that one should only listen to the opinions of the wise.
- He personifies the Laws of Athens to argue that he has an implicit social contract with the city, and to escape would be an act of injustice against the laws that have protected him.

**2. Plato, The Republic, Books I & II**

Book I
- Socrates debates the meaning of justice with his friends.
- He refutes the definitions of justice given by Cephalus (paying debts) and Polemarchus (helping friends and harming enemies).
- He confronts the Sophist Thrasymachus, who claims “justice is nothing else than the interest of the stronger.”
Book II
- Glaucon challenges Socrates to prove that justice is good for its own sake using the story of the Ring of Gyges, which grants invisibility.
- Socrates proposes they define justice in a city first, then apply the definition to an individual, setting up the central metaphor of the book.

3. Sophocles, Oedipus the King and Antigone

Oedipus the King
- Oedipus, king of Thebes, vows to find the murderer of the previous king, Laius, to end a plague.
- He consults the prophet Tiresias, who reveals that Oedipus himself is the murderer.
- Through a series of revelations, Oedipus discovers he has unknowingly killed his father and married his mother, fulfilling a prophecy.
- His mother/wife Jocasta kills herself, and Oedipus blinds himself and goes into exile.
Antigone
- Antigone’s brothers die fighting each other. Creon, the new king, decrees that one will be buried while the other is left unburied.
- Antigone defies Creon’s law, believing her duty to the gods is a higher law than the state’s.
- Creon sentences her to be buried alive.
- The prophet Tiresias warns Creon that his pride will lead to tragedy.
- Antigone, Creon’s son, and Creon’s wife all die, leaving Creon alone with his grief.

**4. Aristotle, Nicomachean Ethics, Book I**

The Highest Good: Aristotle argues that all human actions aim at some good, with the highest good being happiness (eudaimonia), which is pursued for its own sake.
The Function of Man: Happiness is an “activity of the rational soul in accordance with virtue.” To be truly happy, a person must use their unique human capacity for reason.
Virtue and Habit: He distinguishes between intellectual virtue (gained through teaching) and moral virtue (gained through habit).
External Goods: He acknowledges that external goods like wealth and friends are necessary as instruments for a happy life, but not as ends in themselves.

**5. Aristotle, Politics, Book I**

The Nature of the State: Aristotle argues that the city-state is the highest form of human association, formed to achieve the highest good.
“Political Animal”: He introduces the idea that “man is by nature a political animal” and that the city-state is a natural development from the family and village.
Household Management: He examines the three relationships within a household: master and slave, husband and wife, and parent and child, arguing that understanding these is key to understanding the state.
On Slavery and Acquisition: He discusses the concept of “natural slavery” and contrasts the “art of acquisition” for a household with the unnatural pursuit of wealth for its own sake.

6. Plutarch, The Lives

Lycurgus: He established a new constitution for Sparta, including a system of equal land distribution, the agoge (a rigorous education), and a focus on simplicity and civic virtue.
Numa Pompilius: The second king of Rome, he ruled peacefully by focusing on religious and legal reforms, instilling a sense of piety in the Roman people.
Alexander: Tutored by Aristotle, he conquered the Persian Empire and created a vast Hellenistic empire, remembered as a brilliant but complex leader.
Caesar: Plutarch chronicles Caesar’s rise to power, his military conquests in Gaul, his civil war, and his eventual assassination, which ended the Roman Republic.

**7. The Old Testament, The Book of Job**

The Wager: God allows Satan to test the piety of Job, a righteous and wealthy man.
The Affliction: Job loses his children, wealth, and health but refuses to curse God.
The Friends’ Challenge: Three friends visit him and argue that his suffering must be a punishment for some sin. Job maintains his innocence and questions God’s justice.
God’s Response: God appears in a whirlwind and speaks of his own power and wisdom, humbling Job, who accepts his own limited understanding.
Restoration: God restores Job’s health and wealth, and gives him a new family.

**8. St. Augustine, The Confessions, Books I–VIII**

Spiritual Journey: Written as a prayer to God, Augustine’s work is a spiritual autobiography recounting his early life and struggles.
Early Sins: He reflects on the nature of evil, confessing to minor sins of his youth, and sees his own lusts and worldly ambitions as a primary barrier to his faith.
Intellectual Crisis: He recounts his long adherence to the Manichaean religion and his eventual embrace of Neoplatonism, which helped him see evil as the absence of good.
The Garden Conversion: His spiritual journey climaxes in a garden when he hears a voice telling him to “Take up and read!” He opens a book of Paul’s Epistles and is immediately convinced to give up his sinful life.

9. Montaigne, The Essays

The Essay: Montaigne pioneered the essay genre, writing personal, reflective pieces that explore a wide range of topics.
Self-Examination: His main focus is on the study of the self, reflecting his skeptical motto, “Que sçay-je?” (“What do I know?”).
Human Nature: He presents a complex view of human nature, critiquing human arrogance and advocating for humility and a healthy skepticism.
On Death: He famously argues that to philosophize is to learn how to die and that facing one’s mortality can help one to live more fully.
Specific Essays:
- “Of Pedantry”: Montaigne criticizes the pedants of his time for valuing book learning and memorization over true wisdom and judgment. He argues that knowledge is useless if not integrated and applied to life.
- “That It Is Folly to Measure Truth and Error by Our Own Capacity”: He argues against intellectual arrogance and dogmatism, suggesting that we should not assume our own limited reason is the measure of all things. He emphasizes the vastness of the unknown and the diversity of human beliefs.
- “Of Cannibals”: He uses the example of a “savage” tribe to critique the barbarism and ethnocentrism of European society.
- “That the Relish of Good and Evil Depends in a Great measure upon the Opinion we Have of Them”: Montaigne explores the power of our minds to shape our experience of the world. He argues that our opinions, attitudes, and imaginings have a greater impact on our happiness and suffering than external events themselves.
- “Of the Education of Children”: He advocates for an education that emphasizes reason and judgment over rote memorization and encourages exposure to different cultures.

10. Shakespeare, Hamlet

The Plot: The ghost of Prince Hamlet’s father appears, revealing he was murdered by his brother, Claudius, who has now married the queen and seized the throne.
Feigned Madness: Hamlet feigns madness to investigate the crime, leading to a series of tragic misunderstandings and events.
The Play-Within-a-Play: Hamlet stages a play to confirm Claudius’s guilt.
Tragedy: His indecision and delay lead to the deaths of Polonius, Ophelia, Gertrude, Laertes, Claudius, and himself in the final duel.

11. John Locke, Concerning Civil Government

State of Nature: Locke argues that in the state of nature, individuals are free and equal, with natural rights to life, liberty, and property.
Social Contract: To protect these rights, people enter a social contract to create a government.
Consent and Tyranny: Legitimate government is based on the consent of the governed. If a government becomes tyrannical and infringes on these rights, the people have a right to revolt.
Separation of Powers: He advocates for a separation of powers, primarily between the legislative and executive branches, to prevent tyranny.

12. Jonathan Swift, Gulliver’s Travels

Satire: The book is a satirical critique of human nature and European society.
The Voyages:
- In Lilliput, a land of tiny people, Swift satirizes the pettiness of political and religious conflicts.
- In Brobdingnag, a land of giants, Gulliver is small, and the king’s disgust at English society highlights its moral corruption.
- In Laputa, a flying island populated by people obsessed with mathematics and music but lacking common sense, he satirizes abstract and impractical thinking.
- In the land of the Houyhnhnms, rational horses rule over the brutish, human-like Yahoos. Gulliver becomes so disgusted with humanity that he longs to be a horse.

13. Edward Gibbon, The Decline and Fall of the Roman Empire

The Scope: The reading selection focuses on Gibbon’s controversial analysis of Christianity’s role in the fall of the empire.
Chapter 15: The Progress of the Christian Religion: Gibbon investigates the “five secondary causes” for the rapid spread of Christianity, beyond divine intervention. These include the zeal of the early Christians, their doctrine of a future life (including the concept of heaven and hell), the claim to miraculous powers, the pure and austere morals of the believers, and the tight-knit union and discipline of the Christian Church.
Chapter 16: The Conduct of the Roman Government toward the Christians: This chapter details the various persecutions of the Christians. Gibbon argues that the Roman government’s intolerance was rooted in the Christians’ refusal to participate in the traditional pagan rites, which the Romans viewed as a civic duty. He contrasts the Roman policy of general religious tolerance with their harshness toward the Christians’ “inflexible obstinacy.” He also controversially estimates the number of Christian martyrs to be far lower than traditional Christian accounts, and attributes their willingness to face death to fanaticism rather than pure faith.

14. American State Papers and The Federalist

Declaration of Independence: This document asserts the unalienable rights to “Life, Liberty and the pursuit of Happiness,” and states that governments derive their power from the consent of the governed.
The Constitution of the United States of America: This establishes the framework for the U.S. government, including the three branches of government (legislative, executive, and judicial), the system of checks and balances, and the division of powers between the federal government and the states.
The Federalist Papers: A collection of 85 essays written by James Madison, Alexander Hamilton, and John Jay to persuade citizens to ratify the U.S. Constitution.
- No. 1 (Hamilton): An introduction to the series, outlining the need for a new Constitution and the topics to be covered.
- No. 2 (Jay): Argues that the United States should remain a single nation and not be broken up into separate confederacies.
- No. 3 (Jay): Argues that a united nation is better equipped to protect against foreign threats and avoid wars.
- No. 4 (Jay): Continues the argument from No. 3, focusing on the ability of a strong central government to raise and maintain a unified military.
- No. 5 (Jay): Argues that if the states were divided into separate confederacies, they would likely fall into conflict with each other.
- No. 6 (Hamilton): Warns against the potential for civil war and disputes between the states if they do not unite under a strong government.
- No. 7 (Hamilton): Discusses specific causes of conflict among the states if they were not united, such as disputes over territory and commerce.
- No. 8 (Hamilton): Argues that a standing army, which would be necessary for separate states, poses a greater threat to liberty than a unified federal government.
- No. 9 (Hamilton): Advocates for a strong union as a safeguard against domestic faction and insurrection.
- No. 10 (Madison): Famously argues that a large republic is the best way to control the negative effects of factions.
- No. 15 (Hamilton): Outlines the fundamental weaknesses of the Articles of Confederation and the need for a stronger federal government with the power to enforce its laws directly.
- No. 31 (Hamilton): Argues that the federal government must have an unlimited power to tax and raise revenue to function effectively.
- No. 47 (Madison): Defends the separation of powers by citing Montesquieu and clarifying that the branches are not entirely distinct but have a partial agency in the others’ affairs to prevent tyranny.
- No. 51 (Madison): Explains the system of checks and balances and argues that “ambition must be made to counteract ambition.”
- No. 62 (Madison): Describes the Senate, its purpose as a check on the House of Representatives, and the qualifications and election process for senators.
- No. 63 (Madison): Continues the defense of the Senate, arguing for the necessity of a stable body that can provide a sense of national character and policy continuity.
- No. 64 (Jay): Explains the Senate’s role in treaty-making, arguing that it is best suited for this task due to its small size and longer terms.
- No. 65 (Hamilton): Explains the Senate’s power to conduct impeachment trials, arguing it is the proper body for such a solemn and political function.
- No. 66 (Hamilton): Defends the impeachment power of the Senate against various objections.
- No. 68 (Hamilton): Explains the process for electing the President through the Electoral College, arguing it protects against corruption and foreign influence.
- No. 69 (Hamilton): Contrasts the powers of the American President with those of the British monarch and a New York governor, showing the president’s powers are more limited.
- No. 70 (Hamilton): Argues for a single, energetic executive as essential for good governance, national defense, and the administration of law.
- No. 71 (Hamilton): Defends the four-year term for the president, arguing it provides stability and allows the executive to act with firmness.
- No. 72 (Hamilton): Justifies the re-eligibility of the President, arguing it encourages good behavior and allows for experienced leadership.

15. Karl Marx and Friedrich Engels, Manifesto of the Communist Party

Class Struggle: The Manifesto presents a history of society as a history of class struggle, simplified under capitalism into two classes: the bourgeoisie (owners) and the proletariat (workers).
The Revolution: It argues that the exploitation of the proletariat will inevitably lead to a revolution in which the workers overthrow the bourgeoisie and seize control of the means of production.
The Program: It outlines a ten-point program for a new, communist society, including the abolition of private property and a heavy progressive income tax.

Notes: The Odyssey - By Homer

Tue, 09 Sep 2025 00:00:00 GMT

Book 1: A Goddess Intervenes

The gods meet on Mount Olympus and discuss the fate of Odysseus, who has been trapped on the island of the nymph Calypso for seven years.
Athena, his patron goddess, asks Zeus to intervene. Zeus agrees and sends Hermes to order Calypso to release Odysseus.
Athena then travels to Ithaca in disguise and encourages Odysseus’s son, Telemachus, to search for news of his father.

Book 2: A Hero’s Son Rises

Telemachus calls an assembly of the Ithacan people to complain about the suitors who are trying to marry his mother, Penelope, and are consuming his household’s wealth.
He announces his plan to sail to Pylos and Sparta to seek news of Odysseus.

Book 3: A Journey for Answers

Telemachus and Athena (still in disguise) arrive in Pylos and are welcomed by King Nestor.
Nestor tells Telemachus about the return journeys of other Greek heroes but has no information about Odysseus’s fate. He advises Telemachus to visit Sparta.

Book 4: The Kingdom of Menelaus

Telemachus and Nestor’s son, Peisistratus, travel to Sparta and are welcomed by King Menelaus and Queen Helen.
Menelaus recounts his own difficult journey home and tells Telemachus that he learned from the sea god Proteus that Odysseus is alive and being held captive on Calypso’s island.

Book 5: Calypso’s Island

Hermes arrives at Calypso’s island and delivers Zeus’s command to release Odysseus.
Calypso reluctantly helps Odysseus build a raft.
After eighteen days at sea, Odysseus’s raft is shattered by a storm sent by the sea god Poseidon, who holds a grudge against him.
Odysseus is rescued by the goddess Ino and eventually washes ashore on the island of Scheria.

Book 6: The Princess and the Stranger

On Scheria, the princess Nausicaa discovers Odysseus.
She is washing her clothes near the shore with her maidens when they find the exhausted and naked Odysseus.
Nausicaa gives him clothing and instructions on how to approach her parents, the king and queen of the Phaeacians.

Book 7: A Royal Welcome

Odysseus is welcomed into the palace of King Alcinous and Queen Arete.
He receives a warm reception but hides his identity, telling them only of his time on Calypso’s island.

Book 8: Games and a Song

The Phaeacians host games in Odysseus’s honor.
A blind bard, Demodocus, sings a song about the Trojan War, and Odysseus weeps at the memories.
Alcinous notices his distress and asks him to reveal his true identity.

Book 9: The Cyclops

Odysseus begins to tell his story. He and his men land on the island of the Cyclops.
They are captured by Polyphemus, the giant son of Poseidon, who eats several of his men.
Odysseus blinds the Cyclops by driving a heated stake into his eye and escapes by clinging to the underside of the Cyclops’s sheep.
As they sail away, Odysseus reveals his identity, and Polyphemus prays to his father, Poseidon, to curse Odysseus’s journey.

Book 10: Circe’s Island

Odysseus and his men land on the island of the enchantress Circe.
She turns some of his men into pigs.
With the help of the god Hermes, Odysseus outsmarts Circe and forces her to turn his men back.
They stay with Circe for a year, and she eventually tells Odysseus he must visit the Underworld to speak with the prophet Tiresias to find his way home.

Book 11: The Underworld

Odysseus travels to the Underworld.
He meets the ghosts of his mother, Anticlea, and fallen comrades like Achilles and Agamemnon.
The prophet Tiresias warns him not to harm the cattle of the sun god Helios and foretells his long, difficult journey home.

Book 12: Sirens, Scylla, and Charybdis

Odysseus and his men sail past the Sirens, whose beautiful songs lure sailors to their deaths. Odysseus has his men plug their ears with wax and has himself tied to the mast so he can hear their song safely.
They navigate between Scylla, a six-headed sea monster, and Charybdis, a whirlpool.
His men ignore Tiresias’s warning and eat the cattle of Helios, angering the gods.
Zeus sends a storm that destroys their ship, and only Odysseus survives, washing ashore on Calypso’s island.

Book 13: Return to Ithaca

The Phaeacians sail Odysseus back to Ithaca.
He is disguised as a beggar by Athena so he can safely assess the situation in his home.
He is reunited with his faithful swineherd, Eumaeus.

Book 14: The Swineherd’s Home

Odysseus stays with Eumaeus, who, unaware of his master’s identity, tells him how the suitors have been harassing Penelope and consuming the household’s wealth.

Book 15: Telemachus’s Return

Athena visits Telemachus and tells him to return home.
Telemachus and Odysseus are reunited at Eumaeus’s hut, and Odysseus reveals his true identity.

Book 16: The Father and Son Reunion

Telemachus returns to the palace, and Odysseus, still in disguise, follows.
The two plan how they will defeat the suitors.

Book 17: The Beggar in the Palace

Odysseus enters his palace as a beggar. He is only recognized by his dying dog, Argos.
He is insulted by the suitors.

Book 18: The Beggar’s Brawl

Odysseus gets into a fistfight with another beggar, Irus, and easily defeats him, earning the respect of some of the suitors.

Book 19: The Scar

Odysseus and Penelope have a long conversation.
While washing his feet, Odysseus’s old nurse, Eurycleia, recognizes a scar on his leg from a childhood hunting accident. She promises to keep his identity a secret.

Book 20: The Looming Battle

Odysseus, unable to sleep, observes the suitors’ actions and grows increasingly angry.
The suitors continue their disrespectful behavior.

Book 21: The Contest of the Bow

Penelope announces a contest: she will marry the suitor who can string Odysseus’s great bow and shoot an arrow through the handles of twelve axes.
None of the suitors can even string the bow.
The disguised Odysseus asks for a turn, and after being mocked, he easily strings the bow and shoots the arrow, revealing his true identity.

Book 22: The Slaughter of the Suitors

With his identity revealed and the bow in hand, Odysseus, with the help of Telemachus, Eumaeus, and a few loyal servants, begins to slaughter the suitors.
It is a bloody and merciless massacre.

Book 23: The Great Test

Penelope, still unsure of the beggar’s identity, devises a test for Odysseus.
She tells him that she has moved their marital bed, which is built from the trunk of a living olive tree.
Odysseus reveals his knowledge of the bed’s unique construction, and Penelope, now certain, embraces him.

Book 24: Peace and Restitution

The ghosts of the suitors travel to the Underworld.
Odysseus goes to see his father, Laertes, and reveals himself.
The families of the suitors rise in rebellion against Odysseus, but the rebellion is stopped by Athena and Zeus.
The epic ends with peace being restored to Ithaca.

Canonical Text History

The canonical text of The Odyssey, like the Iliad, is a reconstructed text based on a critical analysis of various historical documents. These documents are primarily:

Papyri fragments: The earliest surviving fragments of the Odyssey are on papyrus scrolls, some dating to the 3rd century BC. These fragments, found in places like Oxyrhynchus, Egypt, show that the text was relatively stable in antiquity, although some small variations existed.
Medieval manuscripts: These are the most complete and valuable sources for the canonical version of the poem. The most important is likely a 10th-century Byzantine manuscript, which contains the full text and ancient commentaries (scholia) by Alexandrian scholars. Scholars compare readings from these manuscripts to create the comprehensive modern editions we use today.

Notes: The Prince - By Machiavelli

Tue, 09 Sep 2025 00:00:00 GMT

The Prince: A Chapter-by-Chapter Summary

Chapters 1-11: Types of Principalities

Chapter 1: Machiavelli opens by categorizing principalities, or states ruled by a single person, as either hereditary or new.
Chapter 2: He argues that hereditary principalities are easier to rule because people are accustomed to the ruling family.
Chapter 3: This is one of the longest chapters, detailing the difficulties in governing mixed principalities—newly acquired territories added to an older state. He advises that a ruler should either live in the new territory or establish colonies there.
Chapter 4: Machiavelli explains why the kingdom of Darius, conquered by Alexander the Great, did not rebel after Alexander’s death. He distinguishes between two types of monarchies: one where the prince rules with a powerful nobility, and another where he rules as an absolute authority.
Chapter 5: He discusses how to govern states that were previously free. The most secure way is to destroy them, the next best is for the prince to reside there, and the riskiest is to allow them to keep their old laws and just impose a tax.
Chapter 6: This chapter focuses on new principalities acquired by a prince’s own ability and arms. He uses figures like Moses, Cyrus, Romulus, and Theseus as examples, arguing that great difficulty lies in establishing a new order but that once it’s established, it becomes stable.
Chapter 7: This chapter is a contrast to the last, focusing on principalities acquired by the arms of others or by fortune. He uses Cesare Borgia as his primary example, showing that while such a prince may rise quickly, he lacks a strong foundation and is vulnerable to misfortune.
Chapter 8: Machiavelli examines principalities acquired through wicked means or crime. He argues that while a prince may gain power this way, it will not bring him glory or a lasting reputation.
Chapter 9: He discusses civil principalities, where a ruler gains power with the support of the people or the nobles. He argues it’s better to have the support of the people, as they are a more reliable base of power than the fickle nobles.
Chapter 10: This chapter discusses how to measure the strength of a principality, specifically whether a prince can defend himself or needs external help.
Chapter 11: He explains how to govern ecclesiastical principalities (like the Papal States), which he says are the easiest to maintain because they are upheld by religious institutions and tradition.

Chapters 12-14: Military Matters

Chapter 12: This chapter is a sharp critique of different types of military forces. Machiavelli argues that the foundation of any state is a strong military and that a prince must rely on his own troops, not mercenaries. He argues that mercenaries are disloyal and dangerous.
Chapter 13: He extends his critique to auxiliary troops (forces borrowed from another prince), stating that they are even more dangerous than mercenaries. A wise prince would rather lose with his own army than win with another’s.
Chapter 14: The prince’s primary focus should be on military matters. He must constantly study the art of war, both in practice (through hunting and knowing the terrain) and in theory (by studying history and the actions of great commanders).

Chapters 15-23: The Prince’s Qualities and Conduct

Chapter 15: Machiavelli outlines the qualities that a prince should and shouldn’t have. He famously states that a prince must learn “how not to be good” when necessary, as acting virtuously at all times can lead to ruin.
Chapter 16: He explores the tension between generosity and parsimony. He advises that it is better for a prince to be parsimonious (stingy) to avoid burdening the people with taxes.
Chapter 17: This is one of the most famous chapters, addressing whether it’s better to be feared or loved. Machiavelli concludes that a prince should wish to be both but, since it is difficult, it is much safer to be feared. However, he must avoid being hated.
Chapter 18: Machiavelli argues that a prince must be both a lion (to frighten off wolves) and a fox (to recognize traps). He states that a prince should not always keep his word if it puts him at a disadvantage.
Chapter 19: He explains the importance of avoiding contempt and hatred, which are caused by being perceived as greedy, effeminate, or indecisive. A prince should appear dignified, courageous, and resolute.
Chapter 20: This chapter explores various security measures, like building fortresses, and whether a prince should disarm his subjects or sow divisions among them. He advises that fortresses are useful but not a substitute for the goodwill of the people.
Chapter 21: He discusses what a prince must do to gain reputation, such as engaging in great enterprises and being a firm ally or enemy.
Chapter 22: Machiavelli advises on choosing ministers. He states that a good minister must be loyal and honest, and that a prince can tell a minister’s quality by how he acts in the prince’s best interest.
Chapter 23: He warns the prince to avoid flatterers and to choose a few wise advisers who will tell him the truth.

Chapters 24-26: The State of Italy and a Call to Action

Chapter 24: Machiavelli discusses why the princes of Italy have lost their states. He attributes their failures to their reliance on mercenaries, their failure to prepare for adversity, and their lack of a strong connection with their people.
Chapter 25: He explores the role of fortune versus free will. He argues that fortune controls about half of our actions, but that a prince must be prepared to act decisively and take risks to control the other half.
Chapter 26: This final chapter is a passionate plea for a new prince to liberate Italy from foreign powers. He presents a vision for a strong, unified Italian state led by a prince who can apply the lessons of the book.

Canonical Text History

The canonical text of The Prince is not based on a long line of ancient manuscripts like Homer’s epics. Instead, its textual history is relatively straightforward. The text is based on a single, original manuscript from the 16th century.

The Original Manuscript: The book was written by Machiavelli around 1513 while he was exiled from Florence. He wrote it as a gift to Lorenzo de’ Medici, though it was not formally published during his lifetime.
First Published Editions: The first official editions of The Prince were published posthumously in 1532. These early printed books are the basis for all modern editions.
Later Revisions and Editions: While later editions have corrected minor typographical errors and standardized punctuation, the core text has remained largely unchanged since the first publication. The book’s controversial nature led to it being banned by the Catholic Church, but this did not stop its widespread circulation and influence.

Notes: The Good Life - by Robert Waldinger & Marc Schulz

Sat, 06 Sep 2025 00:00:00 GMT

I found the book a bit slow and repetitive. Most of it common sense, but there were some frameworks that might be worth preserving here.

1. Social Fitness Model

List your key relationships (family, friends, colleagues).
Map each relationship on a chart:
X-axis: How often do you interact?
Y-axis: Does this person energize or deplete you?
Identify gaps: Who do you want to see more? Who drains you?
Schedule regular “relationship workouts” (calls, coffees, walks).
Check in with yourself: Are you investing enough in your most important connections?

2. W.I.S.E.R. Model

Watch: Pause and notice your feelings and the situation before reacting.
Interpret: Ask yourself, “What’s really going on? What’s at stake for me and others?”
Select: Consider your options for responding—don’t just react automatically.
Engage: Choose and act on the best response, thoughtfully and intentionally.
Reflect: Afterward, review what happened and what you learned for next time.

3. Attention as Asset

Track where your attention goes each day (work, family, friends, screens).
Prioritize: Decide who/what deserves your best attention.
Create rituals: Set device-free times (meals, walks, bedtime).
Practice “micro-attention”: Give full focus, even for a few minutes, to people you care about.
Regularly review: Are you spending your attention in line with your values?

4. Relationship Keystones

Assess which types of support you give and receive:
Safety & security
Learning & growth
Emotional closeness/confiding
Identity affirmation/shared experience
Romantic intimacy
Practical help
Fun & relaxation
Identify missing keystones in your relationships.
Diversify: Don’t expect one person to meet all needs—nurture a variety of connections.

5. Generosity & Curiosity Spirals

Look for opportunities to help, support, or show kindness—even in small ways.
Express gratitude: Thank people for what they do, big or small.
Ask deeper questions: Move beyond “How are you?” to “What’s something that made you smile today?”
Be curious about others’ experiences, perspectives, and feelings.
Notice how generosity and curiosity create more connection and positivity.

6. Embrace Change & Corrective Experiences

Remind yourself: Your past does not dictate your future.
Stay open to new people, experiences, and ways of relating.
Notice when reality is better than your fears or expectations—let it update your beliefs.
Seek and accept help when needed; offer it to others.
Allow yourself to be changed by positive experiences, even if they’re small.

7. Friendships & Weak Ties

Value all connections: Close friends, acquaintances, even friendly strangers.
Engage in small talk: Smile, greet, or chat with people you see regularly (barista, neighbor, coworker).
Maintain old friendships: Reach out, even if it’s been a while.
Say yes to invitations or opportunities to connect, even if they seem minor.
Recognize that even brief, positive interactions can boost your mood and sense of belonging.

8. Work Relationships

Invest in work friendships: Eat lunch together, chat, or collaborate on projects.
Mentor or be mentored: Share knowledge, ask for advice, support others’ growth.
Acknowledge contributions: Thank coworkers, celebrate team wins.
Look for meaning in teamwork and helping others at work.
Balance: Bring positive energy from work home, and vice versa.

Notes: The Wealth Ladder - by Nick Maggiulli

Tue, 05 Aug 2025 00:00:00 GMT

It is rare these days for me to read a book about money that I like. I have read too many of them, so I am a bit jaded. But this one I liked. Here are my notes.

I loved the salt metaphor, something like: “Salt doesn’t add taste to food; it enhances the existing flavors. Similarly, money doesn’t create happiness; it amplifies the happiness you already have.”

Introduction

Building wealth isn’t just about working harder; it’s about using the right strategies for your situation.
Wealth is best understood as a series of levels, each requiring different approaches.

Chapter 1: Spending up the Wealth Ladder

Spend based on your net worth, not your income, since income can fluctuate.
The “0.01% Rule” suggests you can safely spend 0.01% of your net worth daily without eroding your wealth.
Avoid overspending, but extreme frugality isn’t necessary.
Use your liquid net worth (readily available assets) as a conservative guide for spending.

Chapter 2: Earning up the Wealth Ladder

Income is the foundation of wealth; high income and high wealth usually go together.
Different forms of leverage (labor, capital, content, code) become more important as you move up the ladder.

Chapter 3: Investing up the Wealth Ladder

Those with less wealth tend to own cash, vehicles, and homes; those with more wealth own investments that generate income.
Moving up the ladder means shifting from assets that cost money to those that make money.
Owning a business or stocks is common at higher levels, but ownership alone doesn’t guarantee success.
The key is to gradually increase your share of income-producing assets.

Chapter 4: Level 1 (<$10k) — Atypical Results Require Atypical Actions

Focus on reducing debt and building a financial safety net.
Prioritize developing skills that can increase your income, ideally without taking on more debt.
Rely on relationships for support if needed, and be prepared to reciprocate.
Avoid large financial risks like high-interest debt or unnecessary loans.

Chapter 5: Level 2 ($10k–$100k) — Learn Today, Earn Forever

Identify where your skills, interests, and market demand overlap.
Education that leads to higher earnings is a worthwhile investment at this stage.
Beware of dead-end jobs and missed opportunities.
The mindset: invest in learning now to benefit for a lifetime.

Chapter 6: Level 3 ($100k–$1M) — Just Keep Buying

Research investments that suit you and consider side hustles to diversify income.
Focus on acquiring income-producing assets and creating multiple income streams.
Overspending, especially on big purchases like homes, is a major risk.
Consistency in investing is key; keep building wealth steadily.

Chapter 7: Level 4 ($1M–$10M) — What Got You Here Won’t Get You There

Decide whether to stay at this level or pursue business ownership for further growth.
Continued investment and possibly starting or joining a business are main opportunities (you need equity).
Avoid speculative investments and businesses that are just jobs in disguise (aka they depend on you).
The strategies that got you to this level may not be enough to move higher.

Chapter 8: Level 5 ($10M–$100M) — Only the Paranoid Survive

Decide whether to sell your business, start a new one, or stop climbing.
Scaling or selling businesses is a common path at this level.
Watch out for concentrated investments, personal liabilities, and the psychological challenges of wealth.
Protect your wealth and relationships; risk management is crucial.

Chapter 9: Level 6 ($100M+) — Legacy = Action × Wealth

Focus on safeguarding your wealth and prioritizing non-monetary aspects of life, like relationships and health.
Consider the impact and legacy you want to leave.
Major risks include divorce, lawsuits, and taking unnecessary risks.
At this stage, your actions and how you use your wealth define your legacy.

Chapter 10: How Long Does It Take to Climb the Wealth Ladder?

Wealth accumulation is a long-term process; most millionaires are in their sixties.
Upward mobility is possible, especially over decades, but most people remain in the same wealth level over ten years.
Time and patience are essential; expect gradual progress, not overnight success.

Final Insights

Money can increase happiness, especially at lower levels, but its impact diminishes as wealth grows.
True wealth includes social, mental, physical, and time wealth—not just financial assets.
Balance is key: financial wealth multiplies the value of other forms of wealth, but can’t replace them (salt metaphor).
The journey up the Wealth Ladder is as much about enjoying life and relationships as it is about accumulating money.

Natalie

Fri, 25 Jul 2025 00:00:00 GMT

Natalie is on the sofa of her mountainside villa. Rain hits the window, carried by the wind from the sea. Her shoulders are relaxed; the children’s rooms are quiet.

Her face looks back from the dark glass. Beautiful. She makes herself widen her eyes, part her lips. She knows this, and the weariness that lies underneath. She frowns.

The phone lies on the table in front of her. Her schedule for tomorrow. She rehearses faces, phrases and jokes. They pay after all. It’s only fair.

She hears the spin cycle of the washing machine finishing. Her shoulders stiffen. The phone pings and she picks it up. It’s a message from her husband. There’s music in the background and the sound of people laughing.

“Sorry I forgot to call you. It is going fantastically! Everyone here loves me! It’s going to be huge! I am going to take you and the kids to Hawaii this summer.”

“Jakob, come here!” a voice calls out.

“Gotta go now. Don’t forget the kids’ dentist appointment and the football match this week. Ahh, and my mum wants you to call her. Love ya’. Bye.”

As the message is playing, a forced smile flickers across her face.

The washing machine finishes and plays its little jingle. She tenses her legs to stand up but sinks back down onto the sofa.

She sees the photograph. Old. Yellow edges. A girl dancing on a table, men watching, mouths open. She puts a hand to her mouth and smiles. Her eyes are wet.

The rain is falling hard now. Natalie removes her socks and shivers as her feet touch the floor. She stands, moves to the window and opens it. The sound of the rain and the scent of pine trees fills the room.

– AND –

She walks into the rain and into the night toward the sea.

– OR –

She gets out in the rain. He is there. She takes his hand and brings him inside. They walk to the sofa leaving wet tracks on the floor.

– OR –

She looks into the night and sees the man waiting for her. She steps forward and the water wets her bare feet. She sighs, steps backward and closes the window.

JediMaster: LLM assigns issues to Coding Agent

Thu, 19 Jun 2025 00:00:00 GMT

I have been working on JediMaster: a Python package that uses an LLM to assign issues to the GitHub coding agent.

The idea is to use the LLM to analyze the issue and determine if it is suitable for the coding agent to work on. If it is, the LLM will assign the issue to the coding agent (or just label it).

You can use it either as a script, as a library, or as a set of Azure Functions (i.e., to automatically assign the issues on a timer).

The code is available on GitHub: JediMaster.

Anna and Frank

Sun, 08 Jun 2025 00:00:00 GMT

Dear unknown hero,
I am writing without knowing who you are, but knowing that you are out there fighting for us all. I don't know war, but I know how it feels to be alone, to be afraid without anyone to turn to for help.

My name is Anna, I live in a small village in the south. The war has not reached us yet. The fields are still green, the sun shines and the birds sing. The lilacs are blooming and the trees are full of fruits.

I wish I could show you how beautiful it is here. My words cannot bring you the peace and joy I feel.

I feel secure knowing that you are out there protecting us all. Never forget that you are a hero and you are not alone. We are all there with you in spirit.

Anna

Anna,

I am Frank. I am at the Western front. I am with the 8th Infantry Division.

It was good to receive your letter. Made me want to be there. Nice, the lilacs.

Please write again.

-F

Dear Frank,

It warms my heart that you liked the lilacs. Their scent so sweet and strong fills the air around my house. I wish you could smell them too.

Today I baked a pie with the first apples of the season. It came out well, soft on the inside and crunchy on the outside. I ate it on the terrace, as the sun went down.

I thought of you. How your world looks like now. Maybe you would have liked my pie.

I hope it is ok to write to you like this. I hope you don't mind.

Anna

Anna,

My world has no color. The sky is cement and the ground is mud. Your letter painted it in colors I had forgotten.

Today I had to repair a truck. Transmission problems. My hands are still dirty for it and with little cuts. Not much water around here to clean them.

Your pie sounds good. I would have liked it. I like apples.

Please keep writing.

-F

Dear Frank,

Today I was in the garden. The sun was warm on my back, but a cool breeze made the leaves dance and my skin tingle.

I had a hard time pulling the weeds. Maybe you would have helped me? I could have used your strong hands and hearing your voice would have made the task go faster.

The evening was colder, so I tucked myself in a blanket and sat on the terrace.

I don't know if I should tell you these small things. I hope you don't mind.

Anna

Anna,

A boy died today. He was 19 and a good boy. A sniper got him. We are all sad and angry and tired.  Your letter made me feel better.

My hands are strong. I would have pulled the weeds for you.

-F

Dearest Frank,

I am sorry about the boy. The world is so cruel. I am happy that my letter helped you feel better, even if just a little.

Perhaps I should not say this, as we have never met, but yesterday night I dreamed of you. We were sitting on my terrace as the sun went down. We were not talking, just enjoying the silence, the breeze and the smell of lilacs.

It is not right that you don't know what I look like. I am sending you a picture of me.

Maybe you won't write to me again once you see it, but I hope you do.

Yours,
Anna

Anna,

You are beautiful. I didn't expect that. I am glad you sent me the picture. I keep it in my pocket and look at it when I can.

I don't have a picture to send you. My comrade has drawn a sketch of me. I am sending it to you. It is not as good as your picture, but it is me.

Please keep writing. Your letters give me strength.
-F

Dearest Frank,

You look handsome in the sketch. It arrived all crumpled up but I ironed it and made it flat again. I keep it in a frame on my nightstand.

At night, the wind brings the smell of lilacs through the window. I look at you and imagine you are here with me.

Is it wrong to think of you like this? I don't know. It feels right, safe and warm. I feel like I know you, even if we have never met.

Yours,
Anna

Anna,

You made me remember the smell of lilacs. I had forgotten it. I wish I could smell them with you.

-F

Dearest Frank,

I will send you a pressed lilac flower. Maybe a little smell will reach you through the paper.

I think about your hands. How strong they must be. How rough they must feel on my skin. I wonder if you would like to hold me. I wonder if you would like to kiss me.

Oh gosh. I am sorry. I should not write this. I am sorry if I made you uncomfortable.

Yours,
Anna

Dear Frank,
It has been a while since you last wrote to me. Did I say something wrong? Was I too forward?

I am sorry if I did. I hope you are ok.

Anna

Anna,

Delivery is slow here. I just received your last two letters. I thought you had stopped writing to me. My heart skipped a beat when I saw your name on the envelope.

I always look forward to your letters. Please keep writing.

-F

Dearest Frank,

I felt so lonely without your letters. I am so happy to hear from you again.

You are the only person that can really see me. You make me feel wanted and happy. I feel like I can tell you anything.

I think about you way too much. I wonder if you think about me too. I wonder if you would like to meet me one day.

I promise to keep writing. I will write to you until you tell me to stop.

Yours,
Anna

Anna,
I think about you too. I wonder what it would be like to meet you. I wonder if you would like me.

I don't speak too much but I am a good listener and my feelings are deep.

Perhaps one day we will meet. I would like that.

-F

Dearest Frank,

Today the sun was shining and the sky was blue. I was cooking in the kitchen and I felt like dancing. I imagined you were there with me, holding me in your arms and spinning me around.

I felt your strong hands on my waist, your breath on my neck. I felt so happy and alive.

Please think of me like this. I am here for you.

Yours,
Anna

Anna,

Your letter made me smile. It sent away the dust and the mud. I want to hold you and dance with you.

-F

Dearest Frank,

The nights are getting colder. I read your letters in bed and they warm me up. They make me feel all soft and fuzzy inside.

What is this feeling? Am I falling in love with you? I think I am. I think I love you.

Yours always,
Anna

Anna,

I love you too. I think about you all the time.

I have good news. I will be home in a few weeks. I will come and find you. Wait for me.

-F

The old woman mutters to herself in the kitchen. Her voice high and thin, like a girl.

“Frank, don’t come home. You can’t come home.”

There is garbage all around her feet. The floor is sticky with old food and spilled drinks. There is the carcass of a rat in the corner, eyes wide open, staring at the ceiling.

“Don’t find me. Don’t find me.”

She sits down on the chair in front of the small table. She picks up the pen and starts writing on a piece of paper.

“Dear Frank …”

She finishes the letter, seals it in an envelope and puts it on the side of the table. She looks at it for a moment then stands up and goes back to her cooking.

There is a strong bang from the other room. The wind has opened a window and it slams against the wall: bang … bang … bang.

The old woman straightens her back and grows taller. She starts talking to herself again, but her voice is deeper now, more guttural.

“Anna, why do you say …”

She moves around in the kitchen retracing her steps. She moves fast with a lot of energy. She doesn’t look old anymore. She doesn’t look like a girl now.

Then she stops and sits on the other side of the table. There are pen and paper on that side too. She picks up the pen and starts writing.

“Anna, why I should not come home? …”

She writes for a long time, the hand uncertain, shaking a little. She finishes the letter, seals it in an envelope and puts it on that side of the table.

There is a mountain of envelopes on the table, on both sides. Some are dirty with old food, the newest ones are clean and white.

The old woman stands up and goes back to her cooking. The window in the other room still bangs against the wall: bang … bang … bang.

The butler

Thu, 15 May 2025 00:00:00 GMT

The car arrived at the plantation at midday, which is a time of sweat in the south. The help lined up in front of the house sweat running down their faces and their backs creating darker patches on their white uniforms. The air smelled of banana leaves.

“She the wife?” asked Myrtle, adjusting the headband on her head. Eugene didn’t reply.

The car stopped in the driveway and sprayed dust in the air. The colonel stepped out and opened the door for her and she came out holding her hat for protection against the sun. They walked up the stairs and in front of the help.

“This is the help, dear. Eugene here has been with us for more than twenty years. He is our butler.”, said the colonel.

Eugene looked at the girl, and she was a girl, twenty at most to the colonel’s fifty. Her skin was very white with red patches because of the sun. Her eyes were pale blue and large and with long lashes. She tilted her head to one side and smiled at Eugene. He smiled back.

The colonel introduced the rest of the help bruskly and energetically waving his hands as he talked. Myrtle gave a slight bow, tripped over her feet and Eugene caught her. The colonel regarded them with a frown and continued with the introductions.

He finished. He marched into the house, pulling the girl with him. The help went back to work. It was harvest season. There was much to do. The heat slowed everything.

The night brought cooler air and Eugene went to the library. He was the only one to use it. The colonel could read, but he had no interest in it. The help could not read and at times Eugene read to them but he preferred to read alone.

As he sat with his book he heard a drunken giggling from the corridor and he saw the couple in the doorframe embracing tightly, her wavering sideways because of their weight difference. She stopped and looked at Eugene with curiosity and a frozen smile on her lips.

“Oh, I let Eugene come and read here when he is done with his work”, said the colonel, looking away. Eugene stiffened and the colonel tightened his grip on the girl’s waist and dragged her, frozen smile and all, toward the bedroom.

As the days went by the help started gossiping. There were shouts of pleasure and shouts of pain coming from the bedroom at night. They could hear hard words spoken and they could hear crying. Myrtle said with admiration that the colonel was an animal. Eugene didn’t say anything.

The girl cheeks got redder because of the sun and she was more comfortable in the house talking to the help with authority but also with kindness and respect while the colonel made his rounds around the plantation. They went to the city together and they often had people over for dinner and the men drank whisky in the library while the women sit down and talked in the dining room. The colonel taught her how to ride and left her roses on her bed.

Eugene didn’t like any of it because it was too much and no flame can burn both bright and long. He looked at the girl and hoped she had good bones for what was coming but he doubted it because she was so slight. He thought about Evaline and her big bones and her big ass and how she died in the plantation when their third child was born.

Evaline was strong and she died strongly. She had a solid quality to her as if she occupied more space on this earth that her frame would allow. Evaline was gone and now there was this girl and he hoped she would be ok but he doubted it because she didn’t occupy any space at all.

One night Eugene saw them from the library door and the colonel wasn’t holding her waist quite as tight and they weren’t giggling and Eugene knew what was about to come.

After a while the trips started and the colonel was rarely home. There was always something in the city that required his attention and the plantation overseer was making the colonel’s rounds. Eugene found glasses around the house that smelled of whiskey and he knew the girl was drinking and he knew that she was not ok.

The girl was losing her kindness and she treated the help with little respect and Eugene thought that it was because she had no respect for herself because of the drinking and the loss of her man.

“What are you reading, Eugene?” she asked one night, words coming out of her mouth with fatigue. She took the whisky bottle from the cabinet and two glasses.

“I don’t drink, Madame”, he said hesitantly and she put one glass back reluctantly.

She poured a big one for herself, sat down in front of him, gulped most of it down and started talking: “You know Eugene that I can’t read? No-one taught me. Why would a belle like me read? I can’t read. Is it difficult? Am I still a belle now that I am married? It’s true, I can’t read.”

Her voice slurred and he couldn’t understand all she was saying and he didn’t reply. She gulped the rest of the glass down and her eyes widened with intention.

“Would you read to me, Eugene?”

“I am not a good reader.”

“But you can read, can’t you?”

“Yes.”

“Then read to me!” she spit out, with excessive force.

“What do you want me to read?” he asked.

“Something … something …” and her voice fell to a whisper “Something … you know … to make me better … smarter … I am not smart … They didn’t teach me smartness … just to be a belle … something to make me smart.”

Eugene was uncertain. These were dangerous waters, he thought, my reading to this girl or my refusing to do it. Either way it is unfit for my position. He looked at the girl and her watery blue eyes streaked with small red veins because of the whisky and her lips parted and her slim white arms. He looked at the girl and started reading.

As he read he watched the girl and saw that she was falling asleep. He tried to vary his tone and speak louder to avoid it but there was no avoiding it and she fell asleep on the chair. The nightgown had opened and Eugene could see her white legs coming out of it and he could see the top of her white breast.

He looked away but then looked back at her and his eyes narrowed and he couldn’t understand it because it had been such a long time without it. He shook his head to dispatch the feelings and reclaim clarity. He tried to wake her up but she just shuffled on the chair and her head fell to one side.

I cannot leave her like this but I cannot touch her either. He looked at the open door. What if someone comes in? What if they find us like this? The fear and the embarrassment and the dark room fueled his feelings and he felt hunger and desire for her and didn’t know why because she was so slight.

He stood up and walked to her and closed her nightgown. He picked her up in his arms and carried her to the bedroom. He put her on the bed and covered her with the blanket. He looked at her for a while, then left the room. He went back to the library and tried to read but couldn’t. He looked at the dark window and thought of nothing.

“Can you read to me, Eugene?” she asked the next night.

“I am an old man, Madame. I get tired from reading too much.”

“Please, Eugene. It would mean a lot to me.” she said petulantly.

“Let’s make a deal. You fill up the small glass because I am too old to last until you finish the big one and I will read to you until the glass is empty.”

She widened her eyes in surprise and poured the small glass full of whisky. Eugene started reading and she was very careful not to drink too much.

They went on like this for a while and she asked him if Darcy and Elizabeth were going to end up together and he said she had to wait a little longer and she asked him if they should end up together and he said that it depends on the reader and to be more careful with the word ‘should’.

“Darcy is a fool”, she said.

“Why do you say that, Madame?”

“Because he is proud and arrogant. He thinks he is better than everyone else.”

The girl’s eyelids were heavy and she was fighting to keep them open and the whisky was all gone.

“Good night, Eugene”, she said without looking at him and she went to bed.

Eugene sat in the library for a long time and thought about the girl and thought about the colonel and thought about Evaline and all his thoughts were jumbled and he couldn’t make sense of them.

The colonel was away for a long time and Eugene read to the girl every night and she drank less and less and the kindness and the respect came back to her.

“Why don’t they get together now, Eugene?”

“There are many things in the way, Madame.”

“But they are of no consequence!”

“There are many things in the way and they are of great consequence, Madame.”

“Maybe you are right. Life is not that simple, is it Eugene?”

“Life is not that simple, Madame, and there are things of great consequence.”

One night the girl told Eugene that the colonel was coming back the next day and they had to finish the book before that. Eugene read to her for a long time and she drank many small glasses of whisky and they were both tired but happy that the book was finished.

Before going to bed the girl kissed Eugene on the forehead and said: “Thank you, Eugene.”

Eugene froze because of the audacity of her and because he didn’t expect it and felt his face getting all warm and his stomach turning. The sun was coming up and he heard a rooster crowing and smelt the morning air coming in through the window.

The colonel arrived in a fury of dust, sweat and noise. The business in the city had been good and there was a lot to say and a lot to do. Plans had to be drawn for the harvest and the overseer had to be called and the workers had to be organized. The house filled with shouts and orders and the sound of feet stamping on the floor.

The night came and Eugene was in the library and he saw the colonel and the girl passing by the door. The colonel was holding her waist tight and she was laughing and they were both drunk. He heard the bedroom door slam and the colonel shout and the girl cry. He heard a loud bang and stood up and walked to the door of their bedroom. The door was ajar because they slammed it too hard and through the crack he saw them.

They were on the big bed and the colonel was on top of her and holding her arms down with his hands. The colonel was kissing her neck and the girl’s face was turned toward Eugene and her eyes were closed and her mouth was soft and her body was limp. Eugene stepped back from the door; clenched his fists and walked back toward the library.

He sat down and stared out the window. The night was dark and the crickets were singing. He sat for a long time until no more sounds came from the bedroom. He sat and thought about Evaline and her big bones and the white of the girl’s legs and her soft lips.

He stood up and opened the tall cabinet and took out the rifle. He got the bullets from the drawer and loaded it and got out of the library. The corridor was dark except for the light coming from the bedroom. Eugene walked toward the light, got to the door, stepped inside and stopped in front of the bed.

The lovers slept and the light came from the moon outside the window and the girl’s legs stuck out of the blanket and they were very white.

The help woke up to the loud bang of the gunshot and the dogs barked and the crickets stopped singing. The night went silent.

SandScore: app to track stats of a beach volley match

Tue, 13 May 2025 00:00:00 GMT

I have created a phone friendly application to track the score and stats for a beach volley game. It is here.

Here are the main user features:

Doesn’t need an internet connection. It is a single html file that can be saved on the phone and keeps state in the browser.
Has a Beginner and Advanced mode depending of the level of statistical sophistication.
Tracks serve turns and score.
Keeps a log of all rallies visible to the user.
Display statistics in both an automatic and bespoken way.
Unlimited undo/redo.
Save and load of a match to file.

The stats tracking part is still buggy. If there is interested in it, I can fix it. Email me.

From a technical perspective the main idea is to have a state machine driving everything: from the UI to the user interaction to the help to the stats displaying.

Here is how it looks for the beginner mode (you might have to scroll right to see all properties):

    "Serve": {
        "displayName": "{servingTeam} Serve",
        "transitions": [
            { "action": "Ace", "nextState": "Point Server", "style": "point", "help": "Direct point from serve", "category": "serve", "statTeam": "Serving", "statPlayer": "0" }, // Simplified stat player
            { "action": "SErr", "nextState": "Point Receiver", "style": "error", "help": "Service error", "category": "serve", "statTeam": "Serving", "statPlayer": "0" }, // Simplified stat player
            { "action": "RE1", "nextState": "Point Server", "style": "error", "help": "Reception error by player 1", "category": "reception", "statTeam": "Receiving", "statPlayer": "1" },
            { "action": "RE2", "nextState": "Point Server", "style": "error", "help": "Reception error by player 2", "category": "reception", "statTeam": "Receiving", "statPlayer": "2" },
            { "action": "R1", "nextState": "Attack Receiver", "style": "regular", "help": "Reception by player 1", "category": "reception", "statTeam": "Receiving", "statPlayer": "1" },
            { "action": "R2", "nextState": "Attack Receiver", "style": "regular", "help": "Reception by player 2", "category": "reception", "statTeam": "Receiving", "statPlayer": "2" }
        ]
    },
    "Attack Receiver": {
        "displayName": "{receivingTeam} Attack",
        "transitions": [
            { "action": "Win1", "nextState": "Point Receiver", "style": "point", "help": "Winning attack by player 1", "category": "attack", "statTeam": "Receiving", "statPlayer": "1" },
            { "action": "Win2", "nextState": "Point Receiver", "style": "point", "help": "Winning attack by player 2", "category": "attack", "statTeam": "Receiving", "statPlayer": "2" },
            { "action": "Err1", "nextState": "Point Server", "style": "error", "help": "Attack error by player 1", "category": "attack", "statTeam": "Receiving", "statPlayer": "1" },
            { "action": "Err2", "nextState": "Point Server", "style": "error", "help": "Attack error by player 2", "category": "attack", "statTeam": "Receiving", "statPlayer": "2" },
            { "action": "Blk1", "nextState": "Point Server", "style": "error", "help": "Blocked player 1", "category": "attack", "statTeam": "Receiving", "statPlayer": "1" },
            { "action": "Blk2", "nextState": "Point Server", "style": "error", "help": "Blocked player 2", "category": "attack", "statTeam": "Receiving", "statPlayer": "2" },
            { "action": "Def1", "nextState": "Attack Server", "style": "regular", "help": "Defended player 1", "category": "attack", "statTeam": "Receiving", "statPlayer": "1" },
            { "action": "Def2", "nextState": "Attack Server", "style": "regular", "help": "Defended player 2", "category": "attack", "statTeam": "Receiving", "statPlayer": "2" }
        ]
    },
    "Attack Server": {
        "displayName": "{servingTeam} Attack",
        "transitions": [
            { "action": "Win1", "nextState": "Point Server", "style": "point", "help": "Winning attack by player 1", "category": "attack", "statTeam": "Serving", "statPlayer": "1" },
            { "action": "Win2", "nextState": "Point Server", "style": "point", "help": "Winning attack by player 2", "category": "attack", "statTeam": "Serving", "statPlayer": "2" },
            { "action": "Err1", "nextState": "Point Receiver", "style": "error", "help": "Attack error by player 1", "category": "attack", "statTeam": "Serving", "statPlayer": "1" },
            { "action": "Err2", "nextState": "Point Receiver", "style": "error", "help": "Attack error by player 2", "category": "attack", "statTeam": "Serving", "statPlayer": "2" },
            { "action": "Blk1", "nextState": "Point Receiver", "style": "error", "help": "Blocked player 1", "category": "attack", "statTeam": "Serving", "statPlayer": "1" },
            { "action": "Blk2", "nextState": "Point Receiver", "style": "error", "help": "Blocked player 2", "category": "attack", "statTeam": "Serving", "statPlayer": "2" },
            { "action": "Def1", "nextState": "Attack Receiver", "style": "regular", "help": "Defended player 1", "category": "attack", "statTeam": "Serving", "statPlayer": "1" },
            { "action": "Def2", "nextState": "Attack Receiver", "style": "regular", "help": "Defended player 2", "category": "attack", "statTeam": "Serving", "statPlayer": "2" }
        ]
    },

From a high level perspective, the state machine contains everything needed to run the app: the name of the buttons to display, the next state, the style of display, help message, stat category, stat team and player.

The UI, stats and logic is generated dynamically by walking over the state machine.

As an experiment, I described to an AI agent how I wanted the code to look (i.e., the usage of a state machine) and iterate on it without looking at the generated code while doing it (aka vibe coding). That’s why the stats part is still buggy as the AI has an hard time figuring it out.

A very polite girl met an Ogre in the woods

Wed, 09 Apr 2025 00:00:00 GMT

A very polite girl met an Ogre in the woods. She was gathering mushrooms and herbs for her children’s dinner. He was eating a roast on a stick. It was cold, the forest was covered in ice and the girl was trembling.

“Good evening Mr. Ogre.”

“Evn’ing.”

“Your meat smells delicious.”

“Want a slice? Might warm you up.”

The girl looked doubtfully at the fat drooling down from the stick and answered: “I thank you for your kind offer, but we are vegetarians. No meat for us but I fully respect your choice of -“

“Why then you said it looked delicious?” interrupted the Ogre.

“Just making conversation”, replied the girl condescendingly.

“You lied?” said the Ogre surprised.

“Just making conversation”, she repeated, slowing down her speech as if she were talking to a child.

“You look awfully cold there. Come by the fire. You can snuggle inside my coat and warm up.”

“Ogre!”, exclaimed the girl, “This is not a proper thing to propose to a married young girl, or to any young girl.”

“My body is warm and you are cold. What is there to shout about?”

The girl sighed with exasperation: “What would your wife say if she knew you held a girl in your arms?”

“We are not doing too well with my wife after the Shenabadumsdumla situation.”

“Which situation was that? If you don’t mind me asking.”

“Shenabadumsdumla … she is our neighbour. She has a great ass. We screwed like rabbits for months hiding in her cave, we tried all positions, even …”

“Ogre!” shouted the girl indignantly “You don’t say ‘screw’, you say ‘made love’”

“Isn’t it the same thing?”, asked the Ogre befuddled.

“No! Love is passion and -“

“We had plenty of passion!” laughed the Ogre.

“And you don’t go into details like that”, continued the girl resigned.

“But I think about the details all the time. Imagine that once -“

“Stop there, Ogre! And now you lost your wife. How sad.”

“I have not lost her. She is still in the cave. She’s just upset. She’ll get over it.”

The girl was losing her cool and was forgetting her ‘please’ and ‘thank you’.

“You are so beautiful” said the Ogre out of the blue.

“I thank you for your kind remark, but I would like you to praise my intelligence instead”, replied the girl, finally remembering her manners.

“I don’t know if you are intelligent, but sure you are beautiful”, stated the Ogre matter of fact.

“The way you say it … it is like you mean it too much. Our friends also have noticed it but say it more subtly and I like that better. Oh, except when we are drunk …”

“What happens when you get drunk?”

“It is not our fault, it is the wine. It is civilized to behave badly when you are drunk. No-one is perfect!”

“So, what happens?”

“Ok … we talk grossly and misbehave. Once a man pat me on my behind and I wanted to kiss a young fella, but didn’t, or did I?”

“You become Ogres!”, exclaimed the Ogre with satisfaction.

“Don’t say such things. We are civilized people. We know our manners. Our mothers taught us well. Except … maybe … the wine …”, the girl sounded uncertain.

The Ogre was getting bored of all that talking.

“Come by the fire”, he whispered.

The girl made a tentative step forward as she was very cold, but then tightened her vest and stepped backward again.

“Come on!”, said the Ogre, getting upset, “Do you really need alcohol to behave properly?”

“I don’t need anything! Don’t speak to me that way!”, shouted the girl, suddenly alive, “For your knowledge, at the village feast while my husband was home with the kids, I slept with a very nice man …”

“See!” said the Ogre triumphantly.

“It wasn’t like that at all. He was quite nice, gentle, and very polite. I found it quite enjoyable. He asked me what I wanted and it was very civilized. I confessed it to my husband and he said that it was ok because it had been so civilized and the butcher - oh, I shouldn’t have said its name - is a good business partner.”

The Ogre yawned. “Yeah, it doesn’t sound like my thing with Shenabadumsdumla at all. Are you coming by the fire or not?”

The girl stepped forward as she was very cold and the Ogre looked very warm, but then she stopped.

“You are so beautiful”, said the Ogre and the girl smiled.

The last ferry

Thu, 03 Apr 2025 00:00:00 GMT

I have to talk to the school and to the tennis coach after work and need steaks for tonight’s dinner and mum wants to chat about my sister … but, no wait … don’t think like that … first things first … that’s what the book says … first thing, my presentation today … did the Boss like it? He is a smart man, but I wish he could speak more clearly … must be Diana’s graphics … yeah, a bit unusual but come on! … and just today the ferry is late … I am going to be late for work again …

and what the fuck does this guy want?

“You have beautiful hazel eyes.”

What??

“I’m sorry. I didn’t mean to intrude.”

The man was young, embarrassed and went back to his table. Claudia looked at him sternly; thin, nervy, faded jeans, a book on his laps - ‘Siddhartha’ - and he wasn’t young. He had gray in his hair. He wasn’t young, but she ‘felt him’ young.

At work, she wasn’t sure if the presentation went well or badly. Focus, Claudia, focus … what is the problem with you? Go … ask for feedback … smile … smile … too many things going on in my life … need more discipline.

“Do you think I have hazel eyes?” she asked her husband, while cooking the steaks.

“I’m not sure. It’s not even a color. I like your ass more!” and slapped it for confirmation. That night they made love for the first time in weeks and she relaxed into it.

The morning came and it was raining, and she looked for the man on the ferry and he wasn’t there. The day passed and she was not thinking about him. People complimented her for how happy she seemed, but she felt confused.

That night they made love again, but she didn’t like it as before. Her husband didn’t notice. “Quite a record we are setting here”, he said before falling asleep.

In the morning she took a later ferry. The water was choppy and she spilled her coffee.

My wardrobe … nothing wearable … need new clothes, brighter clothes … show off my ass … and need to call John for the contract … isn’t ‘Siddhartha’ for teenagers? … and who wears faded jeans anymore?

That night her husband was all over her, but she said no. He turned to the other side and picked up a book. “I guess the streak is over”, he commented wryly.

On the last ferry of the day she was looking for him. Claudia, please, what are you doing? … Is this desperation? For what? What are you desperate for?

The ferry went through a fog bank, small waves hitting the hull and she saw him walking slowly, coffee in hand, careful with it. She felt a lump in her stomach and breathed deeply to calm herself.

She followed him but stopped because she saw where he was going. There was a woman at his table and the woman took the coffee he was carrying, hands careful to avoid any spillage, a timid smile on her face. He touched her arm and her smile widened with uncertainty and delight.

She is not beautiful. I am more beautiful and I have always been beautiful and she is older than me. I had … have … the most beautiful eyes …

Her throat burned, her fists clasped … but then she felt something leaving her, something that was there all along and wasn’t there anymore and will not come back.

She was free of it and maybe she would make her husband happy again tonight for he is a good man and father and you got to know what is important in life.

From the booth where the couple sat came a crystal laugh. While outside it was dark and foggy, there was light over the booth, and the man was whispering and the woman had beautiful hazel eyes.

The cold touch

Tue, 01 Apr 2025 00:00:00 GMT

She got the call late at night, while reading in bed, his voice flat like an answering machine. She sat up putting her back against the pillow, throat tightening.

“Are you serious?” she croaked.

“Yes,” the flat voice said and hung up.

Tears blurred her vision. Her gaze settled on the open wardrobe. “No!” she said to the room and swang her long legs out of bed. She put on a soft dress, her best boots, combed her hair tight and went into the city to see him.

The rain pounded the roof of the car. She kept steady speed to calm down, thoughts flitting through the car’s cabin like fireflies, head tilted sideways doubtfully.

It must be a misunderstanding. Maybe she laughed too much at the boys’ jokes. She should be sexier. She should be more mysterious. Like her sister. Wear high heels.

She got there, but he wasn’t home. He wasn’t at their usual place by the river. She hunkered down in the car. It was cold and she started shivering. She turned on the engine to warm up, but felt guilty about running it still. She turned it off and crossed her legs to keep them warmer. The rain kept falling and she was tired.

They had meet at Jim’s birthday party two weeks earlier. They talked about stars, beers in hand, eyes uncertain on the right procedure. He was tall with a cloud of curly black hair. He moved his hands theatrically, sometimes touching her arm for emphasis or rolling his eyes. His laugh was strong and warm.

They made love that night. He was different from the other guys she used to date. He wasn’t hungry for her. He didn’t lose control. He went about it slow and calm, like an artist painting a difficult subject. She tried to unsettle him with her body and with her mouth, his calmness getting on her nerves. It could all be done very deliberately, she discovered. She didn’t know it could be like that.

The days that followed were the best.

He called her every night. “How was your day? Who did you meet? I thought about you all day.” His voice warm and soft like honey liquor. She sat in bed, legs curled, caressing her nightgown, eyes closed, drinking his voice.

That week end, they went dancing. He danced energetically, muscles bulging, eyes darting like an animal. He attracted attention from the girls and anger from the boys. She stood beside him, spectator to his show.

After, they sat on red sofas in the dim lights of the disco, drinking rum and coke. She kissed him and hugged him and touched him. He stiffened first and then assaulted her with strong hands, pushing his body against hers. She pulled away from him then, ashamed of other people’s eyes.

When they talked, anything she said was a revelation to him, brown eyes boring into hers, head nodding in agreement. He laughed off any disagreement, any tension. His eyes were just for her. Not even a glance to the other girls. They were not worth his attention. Only she was.

At her birthday, he showed up with a single red rose, dressed in a formal shirt and bright shoes. His laugh filled the room as he mingled with all her friends and they all commented on her new boyfriend. “Charming! Lucky you!” bright smiles and pats on the back. Her mum said that he was very beautiful, and she waited for her to continue, but that was it.

Yesterday, he said that he loved her. He said it with the tone a conspirator revealing a secret, curious eyes fixed on hers. She shifted, shuffled. The reply stayed trapped inside. She awkwardly threw her arms around him, kissing his neck. He patted her head and stroke her hair.

Now she was sitting in the car waiting for him. Humidity fogged the windows. Wet leaves smell inside. She put her hand on the glass to clean out the fog and felt it cold to her touch. She also was cold.

He arrived in his black Mercedes and parked just beside her. She opened the door and got in. The car smelled like laundry and the seats were brown. Hard rock music filled the air and he bent forward to turn it off, his hand brushing on her knee.

“Hello,” she said flat. No reply.

She touched his face, shoulders, lower toward his pants. He grabbed her hand and held it still. “Doesn’t change anything. We can still see each other, have fun.”

He guided her hand, small in his, down, his forearm strong, tendons pushing out. She heard a siren going off in the distance. A dog barked. She knew. She had always known.

She got out of the car and looked back at him. He was looking at her, curiously, noting. She stopped for a moment, car door open, waiting for him to say anything, but he didn’t. She closed the door slowly, without anger. She had known all along.

Years later in the street of her town she saw him again. He was older now with grey hair and a brown coat. He walked surely, like someone who knows where he is going.

She felt nothing. He nodded. She nodded back.

No money

Thu, 20 Mar 2025 00:00:00 GMT

The sun kisses the gas pump and hits the old man on the bench. His pants are dark with oil and sweat. He breathes slowly. His eyes are closed.

Outside, the desert.

The car arrives, old and rusty, and the couple gets out in a hurry. They are young; eyes red, skin burned by the sun; torn clothes.

The boy starts filling up the gas tank. The girl gets close to the old man and touches his arm.

“No money” she whispers.

The old man stands up. He towers over the girl for a moment, then shuffles inside the shop, knees popping as he walks.

When the boy finishes, the old man comes back and holds out his hand. The sun catches something metal.

The boy steps backward. The girl’s eyes widen. The wind whistles through the cactuses.

But now the girl sees. She takes the bag: sandwiches and water. She nods her thanks - eyes fill with tears.

The old man sits down again. His breath is shallow. He closes his eyes.

Outside, the desert.

L'uomo con un antenna enorme 1 - Pasquale

Tue, 18 Mar 2025 00:00:00 GMT

C’era una volta un tipo che aveva un’antenna enorme sul tetto di casa. La gente pensava che la usasse per parlare con radioamatori in tutto il mondo, ma non era così. La realtà era ben diversa.

Il tipo in questione si chiamava Alfredo e veniva da una famiglia agiata della zona che però era andata in disgrazia. Gli erano rimasti abbastanza soldi per non dover lavorare, ma non abbastanza per vivere bene. Faceva infatti una vita oziosa e riservata, sempre chiuso in casa a fare chissà cosa.

I vicini pensavano che fosse una spia Russa del KGB. Il verduriere all’angolo giurava di averlo visto insieme ad un noto politico di sinistra in un ristorante della zona. Insomma, nessuno sapeva nulla, se non per il fatto che fosse un radioamatore; fatto su cui erano tutti d’accordo.

E avevano torto. L’antenna di Alfredo serviva ad altro. Era infatti un apparecchio di grado militare costosissimo che lui aveva acquistato alla fine della guerra fredda e speso anni ad apprenderne il funzionamento.

Uno zingaro Rumeno gli aveva anche tradotto il manuale dal Russo, anche se la traduzione risultava spesso incomprensibile. “Inserire il fustacchio nella bobina magnetica fino allo srotolamento del pennacchio”. Immaginati un manuale dell’Ikea di 1538 pagine tradotto dal Russo da uno zingaro Rumeno …

Ma lui aveva perseverato ed alla fine ce l’aveva fatta. Finalmente funzionava tutto. Alfredo poteva sentire le conversazioni telefoniche di tutti gli abitanti del paese. E non solo questo. Vedeva anche tutti i messaggi e poteva intervenire nelle conversazioni. Poteva inviare messaggetti che parevano provenire da un utente in particolare.

Insomma, controllava completamente il flusso dell’informazione.

C’era però un problema di volume: troppe cose da ascoltare. Servivano dei filtri. Primo, eliminare tutte le conversazioni degli adolescenti. Cosa te ne frega di quello che dicono? Secondo, registrare solo le discussioni interessanti. Ma come definire ‘interessanti’?

Alfredo si affido ad un ‘hacker’ Islandese che creò un programma di riconoscimento vocale che ascoltasse tutto, ma registrasse solo le conversazioni contenenti alcune parole chiave, con opportune variazioni. Per esempio, lui inseriva ‘Mangiare’ ed il programma registrava tutte le conversazioni con ‘mangiato’, ‘mangiai’ etc…

Ci volle un po a perfezionare il sistema, ma alla fine tutto funzionava. Al termine della giornata gli arrivava un email con tutte le conversazioni e messaggi interessanti che poi lui si ascoltava in tutta calma. Alfredo era diventato onnisciente.

Ma cosa farsene di tutto questo ben di Dio? Sicuramente c’era una ragione. La provvidenza non gli aveva dato tutto questo potere per nulla. Non poteva sprecare questo dono ascoltando passivamente. Doveva agire; influenzare il corso degli eventi. Ma come farlo? E con quale intento?

Alfredo decise di cominciare con qualcosa di semplice: Pasquale. Pasqù, come era chiamato in paese, era un contadino grosso e rozzo. Aveva fatto solo la prima elementare e non aveva un vocabolario molto vasto. Si faceva bastare una una dozzina di parole, che utilizzava in varie situazioni, affidandosi all’intonazione e ai gesti per variarne il significato. In pratica, risultava incomprensibile.

Lo trovavi al bar del paese ogni venerdì sera a bere vino e straparlare, inanellando queste poche parole in configurazioni inverosimili.

Alfredo decise di accoppiarlo alla professoressa Milani, zitella di lunga data e esperta internazionale dei sonetti in rima alternata dell’Ariosto, di cui ne esistono tre, includendo l’enigmatico: “Io, come il falco, mi astengo”. Quest’ultimo era la causa della notorietà della professoressa che ne aveva proposto un’interpretazione audace. L’ipotesi Milani, come è chiamata, si può riassumere, seppure in maniera incompleta, con la costatazione: “Quando lo ha scritto, era ubriaco”.

Così arrivò il primo messaggio del finto Pasqù: “Oh mia dolce pulzella, mirarti riempe di ardore il mio cuore”.

La professoressa rispose: “Pasquale, mi sa che hai preso un virus”.

Pasqù: “Il mio unico malanno è l’affanno di non poterti avere”.

Milani: “Pasquale, ma cosa dici, hai bevuto?”

Pasqù: “Ti aspetto, ardente come sempre, presso il terzo tavolo dell’amorosa osteria”

Milani: “Amorosa osteria?? Ma intendi il bar Sport?”

Pasqù: “Il dolce luogo del nostro amor”

Milani: “Ma se fa schifo. È pieno di ubriachi!”

Pasqù: “Bacco sarà il nostro chaperone. Se anche tu mi desideri, pronuncia le parole in codice ‘sono la tua anguilla’”

Milani: “Ma che anguilla! Pasqù, ma sei impazzito?” Milani (dopo un po): “Su Pasqù, rispondi, ma ti sei arrabbiato?”

Pasqù arrivò al bar Sport come ogni venerdì sera. Si sedette al suo solito tavolo. Era in gran forma quella sera. Aveva scoperto un nuovo espletivo che ora usava, in forma aggettivale, attaccandolo ad ogni parola. L’aggettivo era ‘straficchioso’. Lo aveva usato un suo amico pastore per descrivere un gallo che si rifiutava di cantare all’ora stabilita. A Pasqù era parso un termine raffinatissimo.

L’ora si era fatta tarda. Le tenebre calavano sul bar Sport. Pasqù, ormai bello ubriaco, cominciava ad assopirsi con la testa sul tavolino. Una musica sdolcinata riempiva l’aria, coprendo le bestemmie del barista che voleva mandare tutti a casa. Il fumo delle sigarette avvolgeva i tavoli creando una nebbia primordiale stile Borneo.

In questo ambiente rarefatto da sogno tropicale arrivò lei. Bella come un confetto. Luccicante come la macchina quando esce dall’autolavaggio automatico. Uscì dalla nebbia come una star di Hollywood: occhialoni da sole, cappello all’Inglese e vestito attillatissimo.

Uno degli avventori, più coraggioso, accennò un fischio di apprezzamento. La prof lo fulminò con lo sguardo in modo tale che l’aria gli rientrò nei polmoni, come se l’avesse rimangiata. Il suono che ne uscì sembrava più un sospiro che un fischio. “Lo scusi professoressa, è un imbecille” disse il barista premuroso.

La Milani si avviò con passo deciso verso il tavolo di Pasquale che, nel suo stato stupefatto, non si era accorto di nulla. Fissandolo intensamente, gli sussurrò, in tono cospiratorio: “Sono la tua anguilla”.

“Non ho capito” rispose Pasqù innocente.

“Sono la tua anguilla” reiterò lei in tono perentorio.

“Ah si?”

“Pasquale, non dire cazzate per favore. Se dico che sono la tua anguilla, vuol dire che lo sono!”

“Se lo dice lei… Cioè tu … o lei … forse.”

“Non ti sarai mica raffreddato verso di me, vero?” ribadì lei stizzita.

“No… No… raffreddato? Io ho sempre caldo… ma in che senso?” rispose lui, ormai nella confusione più totale.

Visto che la situazione non progrediva, la prof cominciò a perdere la pazienza: “Pasqù, per piacere, non fare lo spiritoso. Andiamo via di qui”.

“E dove andiamo?” domandò lui.

“Certo non nella tua cascina, che c’è puzza di capra. Andiamo a casa mia”.

Pasquale, ridendo come un cretino, buttò li una frase a caso composta dalle poche parole a lui conosciute: “Casa è dove la capra dorme”.

La prof spalancò gli occhi, come folgorata: “Pasqù, ma è tuo questo verso?”

“Quale verso?” rispose lui, temendo di avere scorreggiato.

“Ma allora non lo sai neanche di essere un poeta!”

“No” rispose lui sincero.

“Hai qualche altro verso per me?”

Pasqù timidamente improvvisò: “La capra è dove il gallo dorme”.

La prof lo guardò dubbiosa: “Questa è una minchiata, però”

“Scherzavo” buttò li lui, intuendo di avere fallito qualche tipo di test.

“Va beh, andiamo a casa dai. Non si può pretendere troppo da colui che è ignaro dei propri doni”.

“Eh si …” rispose lui incerto.

Alfredo, osservando la scena da uno sgabello all’angolo della sala, sorrideva compiaciuto. Il suo piano stava funzionando.

Candy at the Bar

Fri, 14 Mar 2025 00:00:00 GMT

Candy sits at the counter of the posh hotel bar, eyes fixed on her untouched cocktail; her finger drumming on the dark mahogany. She is dolled up in a short skirt, red lipstick, and dark mascara.

“The first time is a bitch, but you get used to it,” Molly had said. “Most of the clients are good, first class.”

The bar door opens abruptly. Startled, she fixes her gaze on the glass while the barman watches—he knows.

“You have to follow your dream,” her granny used to say. “Whatever the cost. You do what you have to do.”

Her purse hangs from the back of the stool. That’s the signal: red purse. He is going to come to her. No need to look at the door. But still, she looks.

All of a sudden, an old woman takes the stool beside her. “Mum! What are you -” but the images hit her.

Candy sees the inside of a hotel room: her dress on a chair, a pack of cigarettes on a desk. The scene changes: another hotel room, a line of cocaine on the bedside table and muffled laughers.

And it’s all gone. She is in the bar again. She looks for the old woman, but she is not there. Of course she is not there.

The bar door opens. It’s him. Candy knows it’s him; no need to look. She grabs her red purse, hides it behind her back and walks toward the door. He doesn’t even see her. He is scanning the room above her head. She smells cologne on him as she passes by.

And she is out in the street. The smoke from the factory paints the night grey. Horns blare in the air.

Candy spies a waitress in the restaurant on the other side of the road. Her hair is grey and oily. She shuffles from table to table, shoulders hunched, eyes empty, mouth set in a forced grin - perhaps a desperate one.

“What else, Candy?” Molly had said. “What else?”

Candy turns. Her hand rests on the bar door. She pushes it open; music and laughter spill out. Then she withdraws her hand, only to raise it again, trembling, on the handle.

A taxi stops by. A couple gets out. They smile at Candy. She smiles back.

Gino è sparito

Mon, 24 Feb 2025 00:00:00 GMT

Gino era un bambino di otto anni magro magro. Sembrava uscito da un film in bianco e nero del dopoguerra. Era un bambino allegro dal sorriso facile e sdentato. La gente in paese lo conosceva bene e lo fermava per strada per regalargli dolcetti, figurine o anche solo una parola buona. Lui ringraziava timido e ripartiva a razzo verso le avventure che aspettano i bambini troppo attivi.

Io lo ricordo giocare a calcio in piazza mentre noi prendevamo l’aperitivo. “Ciao Gino, tutto bene? Vieni, prendi una patatina”.

Poi, un giorno, Gino è sparito.

Come sempre accade in casi simili, la famiglia fu la prima sospettata. Il padre era un camionista; un lavoro sempre sospetto agli occhi della gente. Poi la madre: troppo bella o forse troppo curata per la sua età. Lo zio ricco che viveva in Germania. Il nonno reduce di guerra ormai semi-rincoglionito. Ma tutte le piste si rivelarono inconclusive. Nessuno sapeva dove era finito.

Passati i primi giorni, le teorie si fecero più audaci. C’era sicuramente un giro di pedofili in paese. O la mafia. Anche il professore di ginnastica fu tirato dentro come possibile amante della madre, o del padre, vista la risaputa omosessualità di entrambi.

Ricordo la signora che fa le carte accusare pubblicamente il verduriere, noto ateo e forse anche comunista - comunque molto antipatico a tutti.

La situazione si chiarificò quando la Pina si presentò in commissariato con passo deciso e si mise ad urlare: “Allora me lo togliete dai coglioni?”

La Pina viveva in una cascina abbandonata ai bordi del paese. Un giorno Gino si era presentato con la sua bicicletta e le aveva fatto una proposta: “Il tuo tetto fa schifo, è tutto rotto. Te lo riparo io se mi compri un gelato.”

Ma il lavoro era lungo e i gelati tanti. Alla fine lei si era stancata di quel bambino che non faceva altro che mangiare a sbafo senza fare nulla per il suo tetto. Così era andata in paese a restituirlo al mittente.

La polizia non sapeva bene cosa fare. Non è che la Pina avesse commesso un vero e proprio reato. A lei quel bambino non era neanche così simpatico. Gino, da parte sua, diceva di essere stato trattato benissimo. La Pina, secondo lui, era un’ottima cuoca e raccontava storie di guerra eccezionali.

Quando gli fu chiesto se non gli fossero mancati i genitori, rispose di no. C’era quel problema del tetto da risolvere …

La signora che fa le carte dichiarò che in fondo aveva ragione lei. La Pina era stata fidanzata con il verduriere in gioventù, quando lei era bellissima e lui bruttissimo, ma molto ricco - e comunque sempre antipatico.

Ho letto ieri sui giornali che la Pina è morta e così mi è tornata in mente tutta la storia. Erano rimasti amici, quei due.

Lui ha continuato ad andarla a trovare in tutti questi anni, malgrado le proteste di lei, a cui non era poi così simpatico.

Non credo che il tetto sia mai stato riparato.

A few good books

Mon, 10 Feb 2025 00:00:00 GMT

I am a voracious reader. Multiple books a week kind of crazy. Occasionally people ask me for a list of good books in the various areas I am interested in. This is it. Just a few books on each topic, perhaps not even the best ones, but the ones that left an impression on me.

Investing for accumulation

In 2012 I have written a post about investing with books recommendations. I mostly stand by it still, with the exception that I now consider dividend investing as a viable alternative even in the accumulation phase. It is less tax-friendly, but the income stream helps you not to panic when the principal decreases in some years.

Investing for income

Once you have accumulated enough assets, you need to generate an inflation-resistant income from them. At this point in time, no insurance company sells inflation-linked annuities, so you are left with two options:

Asset allocation: selling some assets every year hoping that the principal grows enough to replenish what you spend and overcome inflation.
Dividend investing: you live on the increasing dividends from your stock holdings without touching the principal.

I don’t consider real estate investing as passive income. Too much work for me.

On asset allocation, The intelligent asset allocator. On dividend investing, I like anything by Daniel Peris and the more advanced Josh Peters.

Of the two strategies, I am partial to dividend investing, which can be simply implemented in ETF form these days (i.e., SCHD or others).

Classics

In the past few years I have started a classical education on my own. I went to a classical secondary school and was left with an itch to pursue it more. The Great Books of the Western World series is now free on Archive.org. The first few books contain reading plans, but instead I have finished this one. I am currently in the middle of this one, which I like less.

Philosophy

My introduction to stoic ideas came from Irvine. One quickly moves to Seneca’s ‘Letters to Lucilius’ and the rest of the Roman stoic literature, which I prefer to read in Italian (no links).

Management & Leadership

I have been a manager for a good part of my career, but didn’t read too many books about it. Perhaps I considered it as a learn-by-experience kind of thing … A few books I like are Crucial Conversations and Leadership and self deception.

Time management

I had too many emails in my mailbox every day. I needed a system to manage my workload. The philosophy underpinning it is in Getting things done. That is useful in many situations, apart from email overload. The practical implementation is a variation of Total Workday Control.

Statistics

You never understand statistics enough. I don’t remember how I got introduced to the main concepts, maybe at school? A simple tour to uncertainty is Savage.

I am partial to Bayesian statistics, but I learned it directly from a somehow complex book that I really liked: Gelman

I also like Exploratory Data Analysis. The classic is Tukey, but there must be something more modern, certainly in Python …

Health

I encourage everyone to read anything by Gary Taubes. Good Calories, Bad Calories changed my life.

I also read extensively about weight lifting. I think most people don’t realize that you can do it with different goals in mind. Getting bigger is just one of them and perhaps the less important one. Among the trove of crap, there are a few good books.

Because of my age, I am partial to lifting weights slowly, Slow Burn style. Because of my laziness, I like spending the least amount of time doing it as in Body by science.

Coding

I have a post for that here.

Typography

I know, right? Too many good books to mention. Let’s do one: The elements of typographic style.

Narrative

Apart from classics, I read a lot of sci-fi and fantasy. Try Ted Chiang and Patrick Rothfuss. The former is mind opening, the latter writes fantastically (read this).

Others

Many other topics touched my fancy here and there (chess, photography, economy, fonts, design, art history, big history, world politics, …). My library has 100s of books. Mostly I have read, some skimmed, some nothing. Enough for now.

Il quadernetto giallo

Thu, 30 Jan 2025 00:00:00 GMT

AVVISO DI CONTENUTO: linguaggio volgare e contenuti potenzialmente offensivi.

C’era una volta una puttana che amava la matematica. O meglio, amava i numeri. Come qualcuno colleziona francobolli, lei invece collezionava misure.

Visto che era molto bella e ricercata, i clienti la lasciavano fare. A loro pareva quasi un gioco erotico quando lei, magari sul più bello, tirava fuori il righello.

I suoi commenti erano poi ottimisti, ma verosimili. Non è che puoi dire a uno che ce l’ha piccolissimo che è un gigante. Così lei diceva che era ‘sopra la media’. Lo passava alla categoria superiore, per così dire, e loro se ne andavano belli contenti nel corpo e nell’anima.

Però i clienti non sapevano che lei si segnava tutto in un piccolissimo quadernetto giallo con molta precisione: esattamente una pagina per cliente. Per esempio:

NOME: commissario De Vito Andrea
DIMENSIONE A RIPOSO: 20 centimetri
MASSIMA ESTENSIONE: 25 centimetri
CURVATURA MEMBRO: destra
NOTE ADDIZIONALI: carattere nervoso, ma gioviale,
    evidenzia segni di acne giovanile, non menzionare.

A ogni visita, lei aggiungeva qualche dato alla scheda: numero di scarpe, circonferenza del polso … cose così. Inventava anche categorie mai sentite, ma dal suono scientifico: pelosità, coefficiente di mammonaggine … e anche altre misure francamente incomprensibili: ballosità, svergata massima da posizione supina …

Uno può immaginarsi questa piccola scienziata convincere i clienti ad assumere le posizioni più imbarazzanti per ottenere dati molto precisi. Doveva essere molto persuasiva visto che il quadernetto era pieno zeppo.

Come tutto sia venuto alla luce, non è molto chiaro. Lei se ne era già andata da un annetto quando il quadernetto venne pubblicato su un sito web anonimo. L’autore si era anche premunito di fornire liste speciali per facilitarne la consultazione. Così uno poteva più facilmente individuare i “Superdotati”, i “Microdotati”, quelli con una ballosità superiore e così via. Insomma, un vero e proprio catalogo.

Dapprima in paese si cercò di negare. Sicuramente si trattava di uno scherzo. Qualcuno si era inventato tutto. Però c’erano troppi dettagli accurati, troppe informazioni segrete. Per esempio, nessuno sapeva che l’avvocato Caputo avesse un testicolo così pendente che, per comodità, venisse arrotolato intorno all’altro come una liana.

Oppure il fatto che il ragionier Cocchi, quando al culmine del piacere, nitrisse come un cavallo. Per anni il dottor Scalzi aveva cercato nel bosco adiacente quel cavallo maledetto che lo svegliava in piena notte. Alla fine si era convinto che si trattasse di un fenomeno paranormale. Lo aveva in fine registrato e fatto analizzare. “Nessun cavallo nutrisce così” fu il verdetto, che ne confermo l’origine soprannaturale.

Ce ne erano molte di rivelazioni simili. Il quadernetto era autentico.

La seconda fase venne definita ‘la diaspora’. Le mogli cacciarono di casa i mariti infedeli. Tutto cominciò quando la signora Arrighi, che faceva le pulizie al prete, cacciò di casa il suo consorte con le parole minacciose: “Dio ti punirà per ciò che hai fatto!”.

Vista la posizione autoritaria della signora, le altre dovettero fare lo stesso. Lo fecerò però a malincuore. In fondo, non gli e ne fregava molto di essere state fatte cornute.

Quanto ai mariti, passato l’iniziale smarrimento, cominciarono a organizzarsi. Si misero in piedi tornei di biliardo, cirulla e bocce. Ci fu anche una sfida di calcio scapoli contro (ex)ammogliati che si risolse in rissa per un intervento a piedi uniti sul testicolo cadente del Caputo.

I più intraprendenti cominciarono a frequentare il night club locale. Le signorine Lituane non potevano credere alla loro fortuna di fronte a quell’influsso improvviso di babbei. I babbei, a loro volta, non potevano credere che esistessero signorine così belle e gentili.

“Cosa mi sono perso in tutti questi anni!” dichiarò il Cocchi, brandendo felice la sua bottiglia di Champagne da 250 euro. Il cavallo soprannaturale nutrì con molto entusiasmo in quel periodo.

Le mogli, di fronte a questo inaspettato giro di eventi, furono costrette a riprendersi in casa i mariti ora doppiamente fedifraghi. Pareva si divertissero troppo. E poi quanto spendevano!

I mariti, un po a malincuore, tornarono all’ovile. Però non proprio tutti. Un gruppo di furboni si era studiato un piano per continuare ad andare al night club con il benestare delle mogli. L’idea era quella di fingersi afflitti da una malattia molto in voga tra gli attori Americani: la ‘Sex Addiction’. Loro dovevano smettere gradualmente, come Michael Douglas.

Lo schema fallì per due motivi.

In primo luogo ‘Addiction’ non è una parola dalla pronuncia semplice. Malgrado le istruzioni dettagliate sul posizionamento della lingua date dal commendatore Francetti, che si vantava di essere un ‘anglofago’, nessuno ci aveva capito nulla.

Così, sul più bello, si sono tutti incartati. La pronuncia prevalente risultò ‘Addisssion’, con una esse sibilata stile serpente a sonagli. Le mogli, anch’esse non propriamente anglofobe, li fissavano con sguardo interrogativo, così loro dovettero spiegare.

Purtroppo, questa condizione medica, la ‘Sex Addisssion’, suona bene quando la leggi sul giornale, ma quando spiegata si rivela una minchiata totale.

In secondo luogo, ai maschi del paese, questi nomi di attori Americani sembravano tutti uguali. Così ognuno portò ad esempio attori diversi, molti dei quali non afflitti da questa malattia. A confondere ulteriormente le acque c’era il fatto che alcuni degli attori menzionati fossero notoriamente gay. Le moglie pensarono che i mariti stessero confessando una omosessualità nascosta.

L’unico che riuscì a farsi capire fu il Francetti, l’anglofaro, che però si senti rispondere dalla moglie: “Sono così sollevata! Ce la ho anche io l’addisssion! È cinque anni che ti faccio le corna con il Caputo. E io che pensavo di essere una comune troia!” Preso alla sprovvista, lui domandò: “Ma quello con le palle cadenti?”

C’era poi la questione delle misure. Al principio si finse indifferenza. Persino al bar del paese l’argomento non si toccava. Ma questo durò poco. Bastò una sconfitta a cirulla per scatenare il putiferio.

“Con un uccello così meno male che vinci a carte!”. Quella frase ruppe l’incantesimo. Da li in poi si perse ogni ritegno.

Per cominciare furono i nomignoli legati alle misure o a note presenti nel quadernetto. Il barista divenne ‘Minimo’, per ovvi motivi. Il postino, su cui già si sospettava, divenne ‘Gran Minchione’ o ‘Granmi’ per semplicità.

Un povero vecchietto, che girava per il paese con la sua bici scassata, venne rinominato ‘lo Svergatore’ o anche ‘Svergator’, per dirla con il Francetti. Malgrado nessuno sapesse decifrare il significato di ‘coefficiente di svergata’, i giovani del paese presero ad applaudirlo quando passava in bici, facendolo incazzare a più non posso e quasi cadere.

Le donne, inizialmente, si rifiutarono di adottare questi soprannomi vergognosi, ma con il tempo si dovettero conformare all’uso comune. Così per il pese si udivano frasi tipo: “Passo dal minchione a prendere il pane, poi dal moscio per la carne. Spero di incontrare penetrator, che mi deve riparare la calderina”.

Questi nomignoli lentamente persero la loro connotazione originaria e divennero il modo principale di identificare i maschi del paese. “Hai parlato con l’Ambrosi?” “Chi?” “Lo Strizzatette” “Ah, no”.

Alcune donne più intraprendenti cercarono di rendere popolare un soprannome di loro invenzione per elevare lo stato sociale del coniuge. Ma questi sotterfugi non ebbero molto successo, visto che tutti avevano accesso alle misure reali. Ad esempio, la moglie del Minimo provò a dire a una amica “Black ‘n’ Decker”, pronunciato come si scrive, o anche peggio, “vuole le melanzane per cena.” “Chi?” “Mio marito, non lo sai che lo chiamano così?”. E l’amica rispose dubbiosa “Sarà…”

Il quadernetto svelò anche uno dei misteri di più lunga data del paese: perché molte signore si accanissero a farsi fare i capelli da Robertino, malgrado l’evidenza incontrovertibile che, come parrucchiere, facesse proprio schifo. Faceva a tutte una permanente alta e molto voluminosa. Quando si riunivano di fronte alla chiesa sembrava di stare alla corte di Verseille, pronunciato come si scrive, che non è così. If fatto è che, malgrado il nome e la professione. Non era poi gay. Anzi.

A un certo punto, sul sito web del quadernetto comparve un messaggio che diceva: “Se ritenete che la vostra misura sia incorretta, spedite foto comprovante con faccia, membro in piena estensione e righello di riferimento”. “Una specie di selfie con il tuo uccello.” commentò il Francetti.

Malgrado i molti tentativi, nessuno fu in grado di assumere una posizione tale da avere nella stessa foto quelle due parti anatomiche che la natura ha posto così distanti. Si penso quindi che l’autore intendesse una foto frontale a mezzo busto impugnando un righello posto in modo tale da misurare la piena estensione fallica.

Così rassicurato sulle meccaniche dell’operazione, il Minimo pensò a un modo di riabilitare la propria reputazione. Fece giurare al Granmi un voto di segretezza e lo convinse, previa lauta compensazione, a partecipare all’impresa. Codesta impresa consisteva nel trasformarlo, usando baffi finti e parrucca di pelo di cavallo, in una copia del Minimo da fotografare.

Il problema però rimaneva come fargli raggiungere uno stato di massima estensione. Si provò con giornaletti, video e siti web, ma niente pareva stimolarlo. Allora l’Adelina, fedele moglie del Minimo e pia donna di chiesa, sbottò: “Va be’, mi sacrifico io”. E la foto si fece.

L’email fu spedita con titolo conciso: “Correzione misure signor Amadei barista - vedere foto allegata”. La replica arrivo stile telegramma: “Bella foto Granmi in maschera - riservo diritto condivisione sito motociclisti gay”.

Immagino vi chiediate come faccia io a sapere tutte queste cose. All’inizio l’idea era di collezionare dati per la mia tesi di laurea per l’Imperial College of London dal titolo: “Quanto è grosso? A field study of penis dimension of the average Italian male”.

Al sito web ci ho pensato dopo. Lo ho creato per la mia seconda laurea in sociologia. Anche questa tesi fu ben ricevuta, forse anche per il titolo: “A Big Ben? An experiment in radical anatomical transparency.”

Sono pentita di ciò che ho fatto? Alcuni ci hanno un po patito; altri ci hanno guadagnato. Dopo qualche anno non gli e ne fregava più niente a nessuno. Solo i nomignoli sono rimasti.

Il Granmi vive qui con me a Londra. Non ha imparato una singola parola di inglese, ma a me non interessa. Sono altre le cose importanti nella vita.

How I changed my blog (again)

Mon, 13 Jan 2025 00:00:00 GMT

I wanted to try out Cline on a medium-sized project. For a long time I wanted to move away from a static generator like Eleventy to a simple bash or Python script that generates html and css that I can ftp to wherever.

I ended up doing much more than that. All in all I worked on it for a couple of days. Without Cline it would have taken me more than a week, I suspect.

Creating the Python script

I didn’t want to reuse my existing Eleventy partial html fragments and css, so I instructed AI to generate a static website starting from the list of posts in in the posts directory. I then showed AI a picture of my site and asked it to generate a similar one.

We then worked on the list of Code projects generated from the projects.yaml file (I should auto-generate this from specially marked projects on github …). Also, we created the Tags page looking at the front matter of the posts.

After a few iterations, refactorings, and bug fixes, we had a working site similar to the one I had before.

Old site

New site

The new one is slightly simpler graphically, not because I couldn’t do it, but because I chose not to.

This is the 232 lines of Python code that generates the site. To my eyes, the situation is vastly simplified, as I don’t need to understand the intricacies of Eleventy anymore. Everything has become embarrassingly obvious.

Adding a commenting system

In my folly, I don’t want any Javascript on my site.

In the past, I used Disqus, then webmentions, then nothing. But nothing is a bit on the harsh side of life.

I decided to use Github issues as database for the comments. I know about utterances, but I don’t want to use it as it is Javascript based. So I came up with a super simple scheme, where each post has a link to a github query containing the post title. That brings up the correct issue for the post and you can add comments to that.

It is very primitive as you cannot create a comment from the site itself, you need a Github account, etc… etc… But it is simple and it works, no Javascript required.

I also had a bunch of json files with comments from the past. I instructed AI to create a script that automatically generates issues from the posts and comments from the json files. It created another 300 lines of Python (mostly because of the bizarre rate limitations on the Github API).

I haven’t even looked at the script. It is enough that it worked once (took it one night, who knows??).

So now I have imported all the comments from the past and have a working commenting system, for some definition of “working”.

Fixing badly formatted old posts

And now for something that I would have never done without AI. I had a bunch of very old posts that contained very badly formatted and broken code.

The system I used at the time embedded html tags in the markdown. Properly formatting them one by one would have taken a lot of time as there is no automatic way of doing it.

Most of those posts are obsolete, but still I didn’t like them to be in such bad shape. It bugged me.

“AI, buddy, please fix them for me”. And it did. It took a few hours, but it did it. I had to spend a few dollars on consumption, but it worked like a charm.

old post

new post

Out at sea

Sun, 12 Jan 2025 00:00:00 GMT

It had been a long night for the two brothers. They didn’t catch much. They didn’t talk much either.

Gabriel was the oldest. Carlos, the wisest.

That night, it was as if the sea had smashed their faces against the rocks.

“I know about you and Tina,” Carlos started. Gabriel glanced at him, face turning red.

A wave hit the boat. Gabriel swore. They struggled to regain their balance.

Carlos looked down: “This is the last time we ever talk about this.” Gabriel nodded, tears glistening in his eyes.

They pulled up the nets and headed home.

As they reached shore, a wave hit the boat. Gabriel swore. The sun came out and turned the night red.

Edited by: Maia

The sea calls

Sun, 29 Dec 2024 00:00:00 GMT

The beach was long and empty. The sea moved like a prairie, and the sky smelled of lavender. From above, the sun cast its warm, indifferent gaze. At the boundary between sand and water stood the little girl. A passing glance might have missed her. Just another log washed out by the sea.

But she was there all right. Perfectly still, her feet in the water, her head tilted, coal eyes gleaming, black hair streaming down her back. There she was, with the seagulls screaming at the sky and tracing geometries above her.

“Pequeña, ven! You have been out there too long.”

As the years go by, the girl thinks of the sea often. It calls her when she walks among the sharp buildings of the city, inside the elegant hotels and restaurants. She hears it when she puts on her high-heeled shoes; as she holds her child; when she makes love to her husband. At times, she forgets, but never for long.

And time will pass. The old girl finds herself in a hospital bed. She looks down into the coal eyes of another little girl, so much like her own. She moves aside the girl’s long hair and whispers in her ear what the sea taught her all those years ago.

Edited by: Maia

Volley (or racketball or beach-volley) simulation & probability distribution

Sun, 16 Jun 2024 00:00:00 GMT

This is a Jupyter notebook converted to blog post by nbconvert. The notebook is here. To comment or contribute, please open an issue or a pull request on the repository.

The aim is to discover the probability of winning a set/game/match in volleyball/beach volley/racketball/badmington/…, given the probability of winning a point. The model is based on the assumption that the probability of winning a point is constant and independent of the previous points. This is a simplification, but it is a good starting point to understand the game.

Later, we will derive analytical formulas and compare the results of the simulation with the analytical formulas.

Let’s start with the simulation.

import random
import pandas as pd

def simulate_volley(a_win_prob=0.52, max_games=1000, sets_to_win=2, points_to_win_standard=21, points_to_win_last=15):
    """
    Simulate a volleyball match between two teams with given probabilities of winning a point.

    Parameters:
    a_win_prob (float): Probability of team A winning a point.
    max_games (int): Maximum number of games to simulate.
    sets_to_win (int): Number of sets to win the match.
    points_to_win_standard (int): Number of points to win a set unless it's the last set and we are tied.
    points_to_win_last (int): Number of points to win the last set if we are tied.
    """
    a_wins = b_wins = a_tot_points = b_tot_points = a_tot_sets = b_tot_sets = 0
    rows = []

    while a_wins + b_wins < max_games:
        a_sets = b_sets = 0

        while a_sets < sets_to_win and b_sets < sets_to_win:
            a_points = b_points = 0
            points_to_win = points_to_win_standard if a_sets + b_sets < 2 * sets_to_win - 1 else points_to_win_last

            while not (a_points >= points_to_win and a_points - b_points >= 2) and not (b_points >= points_to_win and b_points - a_points >= 2):
                flag = 1
                a_win_point = random.random() < a_win_prob
                if a_win_point:
                    a_points += 1; a_tot_points += 1
                if not a_win_point:
                    b_points += 1; b_tot_points += 1

                if a_points >= points_to_win and a_points - b_points >= 2:
                    a_sets += 1; a_tot_sets += 1; flag = 2
                if b_points >= points_to_win and b_points - a_points >= 2:
                    b_sets += 1; b_tot_sets += 1; flag = 2

                if a_sets == sets_to_win:
                    a_wins += 1; flag = 3
                if b_sets == sets_to_win:
                    b_wins += 1; flag = 3

                rows.append({'F': flag, 'P': int(a_win_point), 'AS': a_sets, 'BS': b_sets, 'AP': a_points, 'BP': b_points,  'AW': a_wins, 'BW': b_wins, 'ATS': a_tot_sets, 'BTS': b_tot_sets, 'ATP': a_tot_points, 'BTP': b_tot_points})

    return rows

Check for convergence

Let’s check that the various probabilities converge to a stable value.

rows = simulate_volley(0.52)

df = pd.DataFrame(rows, columns=['F', 'P', 'AS', 'BS', 'AP', 'BP', 'AW', 'BW', 'ATS', 'BTS','ATP', 'BTP'])
df['WP'] = df['AW'] / (df['AW'] + df['BW'])
df['SP'] = df['ATS'] / (df['ATS'] + df['BTS'])
df['PP'] = df['ATP'] / (df['ATP'] + df['BTP'])

# Avoids SettingWithCopyWarning
pd.options.mode.copy_on_write = True

# Create a dataset containing just rows where Flag is 2
dfp = df[df['F'] == 1]
dfs = df[df['F'] == 2]
dfw = df[df['F'] == 3]

# Plot first N, N1 rows
N = 150
N1 = int(len(dfp['PP']) / 4)
dfw = dfw.head(N)
dfs = dfs.head(N)
dfp = dfp.head(N1)

# Create an increasing integer sequence for x-values
dfw['x'] = range(0, N)
dfs['x'] = range(0, N)
dfp['x']  = range(0, N1)

print(df[['WP', 'SP', 'PP']].tail(1))

# On the same plot, plot the winning probability, set winning probability, and point winning probability in time
import matplotlib.pyplot as plt

fig, ax = plt.subplots(3, 1, figsize=(10, 10))

dfw.plot(x='x', y='WP', ax=ax[0], title='Winning Probability');
dfs.plot(x='x', y='SP', ax=ax[1], title='Set Winning Probability');
ax[2].set_xscale('log')
dfp.plot(x='x', y='PP', ax=ax[2], title='Point Winning Probability');

          WP        SP        PP
92266  0.652  0.609325  0.520164

Simulation

Then we simulate beach volley (2 sets to win, to 21 points each set except the last to 15) and volley (3 sets to win, to 25 each set except the last to 15).

The results look similar to the real data in this blog post

We simulate just in the range of 0.5 to 0.65 for the probability of winning a point, as the probability of winning a set/match become 1 very quickly for higher values.

The results show two things: - the probability of winning a set/match is not linear with the probability of winning a point and it is much higher. I.E. if you win 60% of the points, you win 90% of the sets and much more of the matches. Honestly, I was not expecting this, but it makes sense as the probability of winning a point gets multiplied many times, so small differences end up being very significant. - the longer the set/match, the more the probability of winning a set/match is higher than the probability of winning a point, for the same reason as above (more chances for the better team to win points).

This code can be useful to figure out other characteristics of the game that might be difficult to model with the analytical formulas developed below.

def simulate_volley_prob(low_prob=0.5, high_prob=0.6, step=0.01, max_games=10000, n_sets = 2, points_to_win_standard=21, points_to_win_last=15):
    result = []
    for i in range(int(low_prob * 100), int(high_prob * 100), int(step * 100)):
        rows = simulate_volley(i / 100, max_games=max_games, sets_to_win=n_sets, points_to_win_standard=points_to_win_standard, points_to_win_last=points_to_win_last)
        stats = rows[-1]
        result.append({'Prob': i / 100, 'WP': stats['AW'] / (stats['AW'] + stats['BW']),
                      'SP': stats['ATS'] / (stats['ATS'] + stats['BTS']),
                      'PP': stats['ATP'] / (stats['ATP'] + stats['BTP'])})
    return result

result_2sets = simulate_volley_prob(0.5, 0.65, 0.01, n_sets=2, points_to_win_standard=21, points_to_win_last=15)
result_3sets = simulate_volley_prob(0.5, 0.65, 0.01, n_sets=3, points_to_win_standard=25, points_to_win_last=15)

import matplotlib.pyplot as plt

fig, axs = plt.subplots(1, 2, figsize=(12, 6))

df_2sets = pd.DataFrame(result_2sets)
df_2sets.plot(x='Prob', y=['WP', 'SP', 'PP'], title='Winning Probability 2 Sets', ax=axs[0]);

df_3sets = pd.DataFrame(result_3sets)
df_3sets.plot(x='Prob', y=['WP', 'SP', 'PP'], title='Winning Probability 3 Sets', ax=axs[1]);

plt.tight_layout()

Fitting a function

Here we fit a function to the data in the interval shown. As it turns out, a sigmoid does so pretty well. This is a brute force empirically derived result. Maybe there is a theory behind it, but I don’t know it.

The fit looks pretty good.

import numpy as np
from scipy.optimize import curve_fit
import scipy.stats as stats
import matplotlib.pyplot as plt

def func(x, a, b, c):
    return a / (1 + np.exp(-b * (x - c)))

def fit_curve(df, column):
    popt, pcov = curve_fit(func, df['Prob'], df[column])
    return popt

popt_2sets_WP = fit_curve(df_2sets, 'WP')
popt_3sets_WP = fit_curve(df_3sets, 'WP')
popt_2sets_SP = fit_curve(df_2sets, 'SP')
popt_3sets_SP = fit_curve(df_3sets, 'SP')

fig, axs = plt.subplots(2, 2, figsize=(12, 12))

df_2sets.plot(x='Prob', y='WP', title='Winning Probability 2 Sets', ax=axs[0, 0]);
axs[0, 0].plot(df_2sets['Prob'], func(df_2sets['Prob'], *popt_2sets_WP), 'r-', label='fit: a=%5.3f, b=%5.3f, c=%5.3f' % tuple(popt_2sets_WP))

df_3sets.plot(x='Prob', y='WP', title='Winning Probability 3 Sets', ax=axs[0, 1]);
axs[0, 1].plot(df_3sets['Prob'], func(df_3sets['Prob'], *popt_3sets_WP), 'r-', label='fit: a=%5.3f, b=%5.3f, c=%5.3f' % tuple(popt_3sets_WP))

df_2sets.plot(x='Prob', y='SP', title='Set Probability 2 Sets', ax=axs[1, 0]);
axs[1, 0].plot(df_2sets['Prob'], func(df_2sets['Prob'], *popt_2sets_SP), 'r-', label='fit: a=%5.3f, b=%5.3f, c=%5.3f' % tuple(popt_2sets_SP))

df_3sets.plot(x='Prob', y='SP', title='Set Probability 3 Sets', ax=axs[1, 1]);
axs[1, 1].plot(df_3sets['Prob'], func(df_3sets['Prob'], *popt_3sets_SP), 'r-', label='fit: a=%5.3f, b=%5.3f, c=%5.3f' % tuple(popt_3sets_SP))

for ax in axs.flat:
    ax.legend()

plt.tight_layout()

Fit using a Bernoulli trial model

We can model a set as a Bernoulli trial, answering the question of getting at least $k$ successes before the $r-th$ failure where $k$ and $r$ are the same: the number of points needed to win a set. Similarly, we can apply the same formula to the probability of winning a match given the probability of winning a set.

The approximation works quite well, despite the imperfection of having to win a set by at least 2 points, which is not considered in the analytical formula, but it is in the simulation. My intuition is that once you get to 24-24 and continue playing, the probability doesn’t change.

The formula is taken from here.

It is this:

$$ P = \sum_{j=0}^{r-1} \binom{k-1+j}{k-1} p^k (1-p)^j $$

My intuitive explanation for the formula above follows.

Imagine you want the probability of 3 successes before 3 failures, given the probability of success $p$.

There is just one way to get to 3-0, by 3 successes in a row -> WWW

$ P(3-0) = p^3 $

There are 4 ways to get to 3-1, but WWWL doesn’t count as the match stops at 3-0, hence we are left with LWWW WLWW WWLW. Which can be tought of as ‘all the way to pick 2 wins out out of the first 3 slots, hence $\binom{3}{2}$.

$P(3-1) = \binom{3}{2} p^3 (1 - p)^1$

Equally, you can think at the binomial below as all the ways to get 2 wins in the first 4 slots.

$ P(3-2) = \binom{4}{2} p^3 (1-p)^2 $

The fit is pretty good here too, but now we have a theoretical explanation for it, which is always nice.

from scipy.special import comb

# Range ends in j instead of j-1 as in the formula because the range function is exclusive
def prob_at_least_k_successes_before_r_failures(k, r):
    return lambda x: sum([comb(k - 1 + j, k -1 ) * x**k * (1 - x)** j for j in range(0, r)])

prob_at_least_21 = lambda x: prob_at_least_k_successes_before_r_failures(21, 21)(x)
prob_at_least_25 = lambda x: prob_at_least_k_successes_before_r_failures(25, 25)(x)

prob_win_2sets = lambda x: prob_at_least_k_successes_before_r_failures(2, 2)(prob_at_least_21(x))
prob_win_3sets = lambda x: prob_at_least_k_successes_before_r_failures(3, 3)(prob_at_least_25(x))

fig, axs = plt.subplots(2, 2, figsize=(12, 12))

df_2sets.plot(x='Prob', y='WP', title='Winning Probability 2 Sets', ax=axs[0, 0]);
axs[0, 0].plot(df_2sets['Prob'], prob_win_2sets(df_2sets['Prob']), 'g-', label='P(win 2 sets before opponent wins 2 sets)')

df_3sets.plot(x='Prob', y='WP', title='Winning Probability 3 Sets', ax=axs[0, 1]);
axs[0, 1].plot(df_2sets['Prob'], prob_win_3sets(df_2sets['Prob']), 'g-', label='P(win 3 sets before opponent wins 3 sets)')

df_2sets.plot(x='Prob', y='SP', title='Set Probability 2 Sets', ax=axs[1, 0]);
axs[1, 0].plot(df_2sets['Prob'], prob_at_least_21(df_2sets['Prob']), 'g-', label='P(>=21 successes before 21 failures)')

df_3sets.plot(x='Prob', y='SP', title='Set Probability 3 Sets', ax=axs[1, 1]);
axs[1, 1].plot(df_3sets['Prob'], prob_at_least_25(df_3sets['Prob']), 'g-', label='P(>=25 successes before 25 failures)')

# show labels
for ax in axs.flat:
    ax.legend()

plt.tight_layout()

A linear approximation

In the more reasonable range, 0.50 - 0.55 a linear approximation works pretty good. So, if you are tight on resources, you can use this approximation without too much loss of fidelity.

# Take just the rows with probability less than 0.56
df_2sets_low = df_2sets[df_2sets['Prob'] < 0.56]
df_3sets_low = df_3sets[df_3sets['Prob'] < 0.56]

# Do a linear fit on the data and plot the data and the fit
from scipy.stats import linregress

def fit_linear(df):
    slope, intercept, r_value, p_value, std_err = linregress(df['Prob'], df['WP'])
    return slope, intercept

slope_2sets, intercept_2sets = fit_linear(df_2sets_low)
slope_3sets, intercept_3sets = fit_linear(df_3sets_low)

print(slope_2sets, intercept_2sets)
print(slope_3sets, intercept_3sets)

fig, axs = plt.subplots(1, 2, figsize=(12, 6))

df_2sets_low.plot(x='Prob', y='WP', title='Winning Probability 2 Sets', ax=axs[0]);
axs[0].plot(df_2sets_low['Prob'], slope_2sets * df_2sets_low['Prob'] + intercept_2sets, 'r-', label='fit: a=%5.3f, b=%5.3f' % (slope_2sets, intercept_2sets))

df_3sets_low.plot(x='Prob', y='WP', title='Winning Probability 3 Sets', ax=axs[1]);
axs[1].plot(df_3sets_low['Prob'], slope_3sets * df_3sets_low['Prob'] + intercept_3sets, 'r-', label='fit: a=%5.3f, b=%5.3f' % (slope_3sets, intercept_3sets))

# show labels
for ax in axs:
    ax.legend()

plt.tight_layout()

6.837142857142849 -2.906399999999996
8.25314285714285 -3.6039333333333294

Conclusion

The probability of winning a set/match is not linear with the probability of winning a point and it is much higher. The longer the set/match, the more the probability of winning a set/match is higher than the probability of winning a point. A sigmoid function fits the data pretty well, but a linear approximation works well in the range 0.50-0.55. A Bernoulli trial model can be used to approximate the probability of winning a set/match given the probability of winning a point.

Display math equations in Azure AI Studio Chat

Fri, 31 May 2024 00:00:00 GMT

Your mileage might vary. This is a dirty trick, likely to break the internet, but it kind of works.

Mathematical equations are displayed in LaTex format, basically unreadable.

I have tried to make GPT-4o to format them in a human readable way in the Azure AI Studio Chat Playground, but failed.

So it is like this:

$$
P(A|B) = \frac{P(B|A) \cdot P(A)}{P(B)}
$$

You instead might want something like this:

The trick is to inject Javascript code that executes on a timer and uses MathJax to transform the Latex code to an image.

There are probably better ways to do it. Here is what I did:

Install Tampermonkey
Click on the extension and pick Create a new script
Paste the script below (modified from this thread)

// ==UserScript==
// @name         ChatGPT LaTeX Display
// @namespace    http://tampermonkey.net/
// @version      0.1
// @description  try to take over the world!
// @author       You
// @match        https://chat.openai.com/*
// @match        https://ai.azure.com/playground/*
// @icon         https://www.google.com/s2/favicons?sz=64&domain=openai.com
// @grant        GM_setValue
// @run-at       document-start
// ==/UserScript==


//Prompt should end with>>> Please format any equations in LaTeX.
//TODO: Intercept WebSocket messages and re-typeset every second until the chatbot is done talking.


(function() {
    'use strict';

    console.log("Getting ready to set up MathJax");



    let tmpMathJax = {
        tex: {
            inlineMath: [['$', '$'], ['\\(', '\\)']],
            displayMath: [ ['$$','$$'], ['\[','\]'] ]
        }
    };

    var prescript = document.createElement('script');
    prescript.type = 'text/javascript';
    prescript.innerText = `
    MathJax = {
        tex: {
            inlineMath: [['$', '$'], ['\\(', '\\)']],
            displayMath: [ ['$$','$$'], ['\[','\]'] ]
        }
    };`;
    prescript.onload = function() {
        console.log("MathJax customizations loaded");
    }
    document.getElementsByTagName('head')[0].appendChild(prescript);



    console.log("Setting up MathJax");

    var script = document.createElement('script');
    script.type = 'text/javascript';
    script.async = true;
    script.onload = function() {
        console.log("MathJax loaded!!");

        setInterval(function() {
            console.log("Re-typesetting...");
            MathJax.typeset();
        },5000);
    }
    script.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-chtml.js";
    document.getElementsByTagName('head')[0].appendChild(script);

})();

Caveats: 1. Running something on a timer is ugly. In this case you have to wait 5 seconds to see the nice formulas. 2. It doesn’t work correctly for some inline formulas.

All in all, it is a dirty hack. Perhaps it could be made ‘more working’, but maybe there is a better way altogheter.

How I changed my blog

Sun, 03 Mar 2024 00:00:00 GMT

My blog has been going for more than 20 years. In this timespan there has been just one change in architecture: going from WordPress to Eleventy + Netlify.

At the time, I was looking for the simplest possible solution. I didn’t go far enough.

For a technical blog like mine, you don’t need a static generator like Eleventy. A bash or Python script is enough - so you are not forced to re-learn the static generator every time you pick up the blog again.

Also, you don’t need a web provider that knows about your static generator, like Netlify and compiles your code. You can compile it on your laptop and ftp the html to wherever.

Also, the html and css I was generating was too complex. * Too many div tags to obtain pixel perfect display. * Server side syntax highlighting creating a tangle mess of tags in the <code> tag. * Frivoulous eye candies like TOC, time of reading, web-mentions, … * Too many css variables and indirection for the sake of modularization.

The last point is interesting. In my css, you could change a few typographic variables (i.e., font-size, measure, …) and the whole site would changes. That sounds fantastic, except that it isn’t. I am not Microsoft/Google/Netflix/…

When I pick up the blog again after a few months, I forget how it all works and end up spending hours figuring it out again. Fixing issues means wading through bogs of indirection: is it a browser bug? My framework? …

This is a lesson I had to learn many times, and I am still learning: radical simplicity means you don’t create a framework. You just code the damn thing.

You can see how it used to look by doing a View source on this.

The new website is still Eleventy + Netlify, but I fixed the HTML + CSS to be radically simple. * Not a single div tag: just semantic tags (article, main, …). * No syntax highlighting. Do you really need it for small code snippets? * No eye candies at all. Just text all the way. * No css variables. Just a trivial stylesheet for the typography.

You can see the difference in the source code here.

A few simplification came about as a bonus. When you radically simplify things, you often get to their hidden core.

I used to have different ‘templates’ for different categories of pages (i.e., post, list of posts, home page, …). I now have just one template. The content in the middle of the page is the only thing that changes.

One thing I have not removed is a processing pass that insert typographic niceties (like Hanging Punctuation). You can see an example on the line that says: “I want a piece …” here.

This is frivolous, but for someone who is into typography like me (I have created my own font after all), it does matter.

The old website was already at 100% in all google lighthouse categories. The new one is even faster. Not that it matters, but you know …

The next step is to remove the dependency from the static site generator and provider. Just a simple script that iterates over all the posts and generates the Notes, Tags, Code pages should be about it. Next time.

Oh, and I also need to re-introduce a way to comment on the posts. I am thinking some kind of public email box with one thread for each post, but I have not investigated further.

The best programming languages to learn in 2024

Wed, 21 Feb 2024 00:00:00 GMT

As few people have asked for my opinion. Here it is in very concise form.

It depends on your goal and it is all very debatable.

To fundamentally understand programming, I like a ‘barbell’ strategy. One language close to the machine and one that is very abstract: * C (the machine) * Haskell (the abstractions)

Bonus shouts to Racket (macros), Prolog (logical programming) and Erlang (concurrency).

The other languages are pragmatic compromises of the fundamental concepts that are learned more purely with the languages above this line.

F# is a good compromise between machine and abstractions and might be a good way to kill two birds with one stone.

To be productive immediately and find (easily?) remote (low?) paying jobs as web developer: * HTML (not really a language) * CSS (not really a language) * Javascript

Bonus shout to Typescript.

To work in the corporate world for an obscene amount of money: * C# * Java

Bonus shouts to Powershell and Bash for automation; Rust to look cool to the hiring manager.

To work as a data scientist (AI AI AI …): * Python * C++

… And a good math background. Bonus shouts to R and Julia.

There are fantastic books on all these languages. Everyone has his favorite.

I also have no idea what is best to teach young people as I have never done it.

They probably prefer to build funny stuff (i.e., games) which is not trivial with my ‘fundamental choices’ of C and Haskell. Compromises need to be made.

I know there is a good Racket book along those lines. Perhaps there are others. I have not investigated the matter.

I am speaking at GoTech World in Bucharest

Mon, 23 Oct 2023 00:00:00 GMT

Come and watch me here

Excel Financial Functions live on

Fri, 05 Aug 2022 00:00:00 GMT

I have a sweet spot for this library.

I coded it the summer my daughter was born in 2008, while back in Italy for three months. It is one of my first forays into functional programming.

I coded most of them with the sea in front of me. I would bet that it is one of the most Excel-compliant libraries in existence across all languages.

I am happy that the tests (all 199,252 of them) still pass! This is a testament to good backward compatibility in the language.

Thanks to James for keeping it alive.

Using Zig comptime for conceptual dryness

Wed, 09 Mar 2022 00:00:00 GMT

On github.

The Problem

While writing my C# Forth, I grew unhappy about the conceptual repetition in the code. To add a new Forth word, you have to add a new value to the enumerator that represents the opcode, add a new member to a hashtable that maps it to a string (what the user types) and finally implement the action for the word. See here.

Granted, by using a struct, I could have kept these items geographically closer, but the repetition is there. Using reflection, I could have removed it, but then performance would suck in the inner loop of my interpreter.

My familiarity with the Zig language made me realize that I could remove the redundancy by using zig’s comptime, without any loss in performance. This post describes a prototype of that.

The Solution

The idea is to have functions on my main struct in the form op_WORDNAME. At compile time, I generate the enumerator with all the opcodes and the series of conditional statements to call the right function op_WORDNAME when the user types WORDNAME. By making the functions inline, I don’t even pay the price of a function call. BTW: Zig gives a compile-time error if it can’t inline.

In the Vm code below, if you want to support a new word, you add a new function. The rest of the code is unchanged.


const std = @import("std");

const Vm = struct {
    state: i32 = 0,

    inline fn op_double(this: *Vm) void {
        this.state = this.state * 2;
    }
    inline fn op_plus1(this: *Vm) void {
        this.state = this.state + 1;
    }
    inline fn op_notFound(_: *Vm) void {
        std.log.info("{s}", .{"Word not found."});
    }
    inline fn op_bye(_: *Vm) void {
        std.process.exit(0);
    }
};

The Shell Loop

This is the shell main loop. It is pretty standard stuff until you get to the innermost while loop, marked with (*). Even if it looks like the code is calling two functions (findToken and execToken), it isn’t. The compiler replaces these two function calls with a series of if statements to match a string with the corresponding opcode and to find the correct function to call.

Even the enumerator Token is not defined anywhere in the code. It is generated automatically at compile time and contains one value for each op_XXX function defined on Vm. Even if I have not defined it, I can still access the values normally (i.e., see Token.ntFound below).


fn shellLoop(stdin: std.fs.File.Reader, stdout: std.fs.File.Writer) !void {
    const max_input = 1024;
    var input_buffer: [max_input]u8 = undefined;
    var vm = Vm{};

    while (true) {
        try stdout.print("> ", .{});

        var input_str = (try stdin.readUntilDelimiterOrEof(input_buffer[0..], '\n')) orelse {
            try stdout.print("\n", .{});
            return;
        };

        if (input_str.len == 0) continue;
        var words = std.mem.tokenize(u8, input_str, " ");

        while (words.next()) |word| { // (*)
            const token = findToken(word) orelse Token.notFound;
            execToken(&vm, token);
        }
        std.log.info("{}", .{vm.state});
    }
}

Finding Tokens

So, how do we do it? Let’s start with findToken. The code is relatively simple because you are not writing a ‘macro’. You are just writing normal Zig code. For .NET programmers, this is like having System.Reflection and System.Reflection.Emit available at compile time.

You could do something similar with C# source generators, but you would need to operate on a rather complex ‘AST’ and generate code using string concatenation. It would probably be dozens of lines of very intricated code. Here, it is three simple lines.

An inline for tells Zig to unroll the loop. The compiler iterates overa all the field of the Enum and generates a series of if statements that return the value of the Enum that matches the given string (what the user typed).


inline fn findToken(word: []const u8) ?Token {
    inline for (@typeInfo(Token).Enum.fields) |enField| {
        if (std.mem.eql(u8, enField.name, word))
            return @field(Token, enField.name);
    }
    return null;
}

Token Execution

Token execution is similar. Again we unroll the loop at compile time, generating a series of if statements that execute the Vm function corresponding to the given Token.


inline fn execToken(vm: *Vm, tok: Token) void {
    inline for (@typeInfo(Token).Enum.fields) |enField| {
        const enumValue = @field(Token, enField.name);
        if (enumValue == tok) {
            const empty = .{};
            _ = @call(empty, @field(Vm, "op_" ++ @tagName(enumValue)), .{vm});
        }
    }
}

Token Generation

We generate the Token enumerator by iterating over all the declarations on the Vm struct and generating a set of declarations that are then use to compile time construct the correct Enum using the @Type builtin function.


const Token = GenerateTokenEnumType(Vm);

fn GenerateTokenEnumType(comptime T: type) type {
    const fieldInfos = std.meta.declarations(T);
    var enumDecls: [fieldInfos.len]std.builtin.TypeInfo.EnumField = undefined;
    var decls = [_]std.builtin.TypeInfo.Declaration{};
    inline for (fieldInfos) |field, i| {
        const name = field.name;
        if (name[0] == 'o' and name[1] == 'p') {
            enumDecls[i] = .{ .name = field.name[3..], .value = i };
        }
    }
    return @Type(.{
        .Enum = .{
            .layout = .Auto,
            .tag_type = u8,
            .fields = &enumDecls,
            .decls = &decls,
            .is_exhaustive = true,
        },
    });
}

Assembly Output

Ok, but does it really work? Well, you can run it with zig build run, but it is really inlining correctly? Well, the assembly language says yes. No calls to external functions in the main loop.

 const token = findToken(word) orelse Token.notFound;
 2310db: f6 85 a1 fa ff ff 01 testb $0x1,-0x55f(%rbp)
 2310e2: 75 09 jne 2310ed <shellLoop+0x38d>
 2310e4: c6 85 9f fa ff ff 02 movb $0x2,-0x561(%rbp)
 2310eb: eb 0c jmp 2310f9 <shellLoop+0x399>
 2310ed: 8a 85 a0 fa ff ff mov -0x560(%rbp),%al
 2310f3: 88 85 9f fa ff ff mov %al,-0x561(%rbp)
 home/lucabol/dev/zig-forth/src/main.zig:65
 execToken(&vm, token);
 2310f9: 8a 85 9f fa ff ff mov -0x561(%rbp),%al
 2310ff: 48 8d 8d 58 fb ff ff lea -0x4a8(%rbp),%rcx
 231106: 48 89 4d d8 mov %rcx,-0x28(%rbp)
 23110a: 88 45 d7 mov %al,-0x29(%rbp)
 execToken():
 home/lucabol/dev/zig-forth/src/main.zig:32
 if (enumValue == tok) {
 23110d: 31 c0 xor %eax,%eax
 23110f: 3a 45 d7 cmp -0x29(%rbp),%al
 231112: 75 50 jne 231164 <shellLoop+0x404>
 home/lucabol/dev/zig-forth/src/main.zig:34
 _ = @call(empty, @field(Vm, "op_" ++ @tagName(enumValue)), .{vm});
 231114: 48 8b 45 d8 mov -0x28(%rbp),%rax
 231118: 48 89 45 e0 mov %rax,-0x20(%rbp)
 Vm.op_double():
 home/lucabol/dev/zig-forth/src/main.zig:7
 this.state = this.state * 2;
 23111c: 48 8b 45 e0 mov -0x20(%rbp),%rax
 231120: 48 89 85 58 fa ff ff mov %rax,-0x5a8(%rbp)
 231127: 48 8b 4d e0 mov -0x20(%rbp),%rcx
 23112b: b8 02 00 00 00 mov $0x2,%eax
 231130: 0f af 01 imul (%rcx),%eax
 231133: 89 85 60 fa ff ff mov %eax,-0x5a0(%rbp)
 231139: 0f 90 c0 seto %al
 23113c: 70 02 jo 231140 <shellLoop+0x3e0>
 23113e: eb 13 jmp 231153 <shellLoop+0x3f3>
 231140: 48 bf 68 1f 20 00 00 00 00 00 movabs $0x201f68,%rdi
 23114a: 31 c0 xor %eax,%eax
 23114c: 89 c6 mov %eax,%esi
 23114e: e8 fd 32 fd ff callq 204450 <std.builtin.default_panic>
 231153: 48 8b 85 58 fa ff ff mov -0x5a8(%rbp),%rax
 23115a: 8b 8d 60 fa ff ff mov -0x5a0(%rbp),%ecx
 231160: 89 08 mov %ecx,(%rax)
 execToken():
 home/lucabol/dev/zig-forth/src/main.zig:32
 if (enumValue == tok) {
 231162: eb 02 jmp 231166 <shellLoop+0x406>
 231164: eb 00 jmp 231166 <shellLoop+0x406>
 231166: b0 01 mov $0x1,%al
 231168: 3a 45 d7 cmp -0x29(%rbp),%al
 23116b: 75 4c jne 2311b9 <shellLoop+0x459>
 home/lucabol/dev/zig-forth/src/main.zig:34
 _ = @call(empty, @field(Vm, "op_" ++ @tagName(enumValue)), .{vm});
 23116d: 48 8b 45 d8 mov -0x28(%rbp),%rax
 231171: 48 89 45 e8 mov %rax,-0x18(%rbp)
 Vm.op_plus1():
 home/lucabol/dev/zig-forth/src/main.zig:10
 this.state = this.state + 1;
 231175: 48 8b 45 e8 mov -0x18(%rbp),%rax
 231179: 48 89 85 48 fa ff ff mov %rax,-0x5b8(%rbp)
 231180: 48 8b 45 e8 mov -0x18(%rbp),%rax
 231184: 8b 00 mov (%rax),%eax
 231186: ff c0 inc %eax
 231188: 89 85 54 fa ff ff mov %eax,-0x5ac(%rbp)
 23118e: 0f 90 c0 seto %al

Also, more directly, see below:

 return @field(Token, enField.name);
 230ff0: c6 85 a1 fa ff ff 01 movb $0x1,-0x55f(%rbp)
 inlined by /home/lucabol/dev/zig-forth/src/main.zig:64 (shellLoop)
 230ff7: c6 85 a0 fa ff ff 00 movb $0x0,-0x560(%rbp)
 inlined by /home/lucabol/dev/zig-forth/src/main.zig:64 (shellLoop)
 230ffe: e9 d8 00 00 00 jmpq 2310db <shellLoop+0x37b>
 home/lucabol/dev/zig-forth/src/main.zig:24
 inlined by /home/lucabol/dev/zig-forth/src/main.zig:64 (shellLoop)
 if (std.mem.eql(u8, enField.name, word))
 231003: 48 8b 85 a8 fa ff ff mov -0x558(%rbp),%rax
 inlined by /home/lucabol/dev/zig-forth/src/main.zig:64 (shellLoop)
 23100a: 48 89 45 b0 mov %rax,-0x50(%rbp)
 inlined by /home/lucabol/dev/zig-forth/src/main.zig:64 (shellLoop)
 23100e: 48 8b 85 b0 fa ff ff mov -0x550(%rbp),%rax
 inlined by /home/lucabol/dev/zig-forth/src/main.zig:64 (shellLoop)
 231015: 48 89 45 b8 mov %rax,-0x48(%rbp)
 inlined by /home/lucabol/dev/zig-forth/src/main.zig:64 (shellLoop)
 231019: 48 bf e0 2a 20 00 00 00 00 00 movabs $0x202ae0,%rdi
 inlined by /home/lucabol/dev/zig-forth/src/main.zig:64 (shellLoop)
 231023: 48 8d b5 a8 fa ff ff lea -0x558(%rbp),%rsi
 inlined by /home/lucabol/dev/zig-forth/src/main.zig:64 (shellLoop)
 23102a: e8 61 ac fd ff callq 20bc90 <std.mem.eql>
 inlined by /home/lucabol/dev/zig-forth/src/main.zig:64 (shellLoop)
 23102f: a8 01 test $0x1,%al
 inlined by /home/lucabol/dev/zig-forth/src/main.zig:64 (shellLoop)
 231031: 75 02 jne 231035 <shellLoop+0x2d5>
 inlined by /home/lucabol/dev/zig-forth/src/main.zig:64 (shellLoop)
 231033: eb 13 jmp 231048 <shellLoop+0x2e8>
 home/lucabol/dev/zig-forth/src/main.zig:25
 inlined by /home/lucabol/dev/zig-forth/src/main.zig:64 (shellLoop)
 return @field(Token, enField.name);

And the driver is obvious.


pub fn main() !u8 {
    const stdin = std.io.getStdIn().reader();
    const stdout = std.io.getStdOut().writer();
    try stdout.print("*** Hello, I am a Forth shell! ***\n", .{});

    try shellLoop(stdin, stdout);

    return 0; // We either crash or we are fine.
}

Horchata

Wed, 09 Mar 2022 00:00:00 GMT

Here is a house on a hill overlooking the highway. Behind that, the sea. The old couple sits on the terrace, watching the traffic snake by. In all these years, they have seen a lot. There have been accidents, a police chase, a shooting even. Once a horse wandered on the road.

“I want a piece of all that life,” says the old man. His name is Pedro and he has the heart of a child. He goes down to the garage and builds a big wooden sign that says “Horchata.” His wife rolls her eyes. Her name is Maite and she is very patient.

The next day, they wake up early. Maite grinds the almonds and mixes the sugar while Pedro puts up the sign. They wait.

People come—first families, then the big trucks. Word goes around and lines start to form. They are longer every day.

It’s not just the horchata. Maite sings sometimes. She has the voice of a hummingbird, but you have to coax it out of her. She is too shy. Pedro talks to the men about sports and politics. He has a lot of opinions.

But it is mainly the horchata that does it. The sea air gives it flavor. A drop of whisky makes it unique.

The day batch finishes in the early mornings. The customers don’t like that and honk their horns to tell the others back in the line. Pedro’s heart sinks. He goes back to the garage and builds another sign that says: “se acabo” - it is finished. They put the sign up every late morning to avoid people getting out of the highway for nothing.

The line of customers starts forming earlier every day. Some sleep the night in line. Pedro brings them coffee in the morning. He feels guilty it has come to that.

They need to increase production - he reasons. But Maite has just two hands and there is just one almond tree. So he puts more water in it. And then some more. He also lowers the price because he is an honest man. The lines get even longer. They still honk when the sign comes up.

Now Maite wants a vacation. She is tired of all the grinding. She doesn’t even like almonds and she is getting addicted to sugar and whisky. Her voice slurs when she sings.

Pedro has an idea. He gets a copyright on the name “Maite’s horchata” and goes and talks to all the bars in town. He lets them use the name and the recipe but takes a percentage of the sales. Maite is a brand.

Sales skyrocket. Some bars don’t follow the recipe, but Pedro is on their case and makes sure quality is preserved. The couple still lives in the house on the hill, but now it is bigger. It got a sign that says: “The original Maite.” They have people making the horchata now. They get the almonds from Marocco. Pedro doesn’t talk to customers anymore. Maite doesn’t sing. Her voice is not what it used to be.

And big money comes. They can go national, but they have to change the recipe. You see, almonds are too expensive and whiskey is a no-no. They use caramel and more sugar instead.

Maite is a celebrity now. She got her face on all these cans. She is always on TV. She tries to sing, but her voice comes out funny. It is all these parties she has to go to. The music is too loud and the smoke gets in her lungs.

Pedro enjoys the attention. He buys a fancy car and drives Maite around town to her appointments. He wears shoes made with crocodile skin and they are very pointy. He talks loud and drinks a lot. Sometimes he falls asleep in the car parked in the driveway to their new villa.

The house on the hill is a museum. They have photos of how things were when it all started. The lines of trucks outside; Maite grinding almonds; Pedro building the big sign.

One night they go to a movie premiere. In the car, they talk about their good luck. Sure there are problems: they are busy, always running around. They are tired. They drink too much. But look at what they have!

And they look. And they look …

… and find nothing of importance.

If you come up the road to the house, you better come up quietly. Don’t make a fuss of it. When you are up on the terrace, don’t make a sound. You might catch Maite singing as she goes about her business. You might wake up Pedro. He will be happy to see you. He enjoys the occasional chat about sport and politics. He has lots of opinions.

In the garden, here is the almond tree. Down below snakes the highway. Behind that, the sea.

Implementing Forth.Net

Mon, 07 Mar 2022 00:00:00 GMT

Abstract

This is a Forth for the .NET framework in one cs file. It is a token threaded implementation that can save its status to a very concise binary format. Feel free to reuse this code as you wish.
## How to use it Simply copy this file or reference the nuget package Forth.Net in your project. This is the public interface:

public class Vm {
    public Vm( ... sizes of memory areas ...);

    public Func<string>? NextLine;
    public void Quit();

    public void EvaluateSingleLine   (string forthLine);
    public void EvaluateMultipleLines(string forthLines);

    public bool Debug;
    public void Reset();

    public IEnumerable<string> Words;
    public string StackString();
}

You set NextLine to tell Forth where to get the next line of input from. Quit starts the interpretation. EvaluateSingleLine and EvaluateMultipleLines are utility functions. You can write them yourself with NextLine and Quit. The rest should be self explanatory.

## Preliminaries The ambition was to write a Forth that can be recompiled for 32 or 64 bits. I just tested the 64 bits part. The rest is just standard using stuff that I can’t move to a global.cs file because I want to be able to simply copy this cs file to a project and have my Forth there.

#if CELL32
using Cell      = System.Int32;
using Index     = System.Int32;
#else
using Cell      = System.Int64;
using Index     = System.Int32;
#endif

using Code    = System.Byte;
using AUnit   = System.Byte;
using AChar   = System.Byte;

using System.Text;
using System.Runtime.CompilerServices;
using System.Buffers.Binary;

using System.Diagnostics.CodeAnalysis;
using System.Reflection;
using static Forth.Utils;

[assembly:InternalsVisibleTo("Forth.Net.Tests")]

namespace Forth;

public class Vm {

## The outer interpret In Forth everything starts at Quit. Yep. It is the line interpreter. It reads a line (with Refill) and interprets it.

    public void Quit()
    {
        rp = 0; Executing = true; ds[inp] = 0; // Don't reset the parameter stack as for ANS FORTH definition of QUIT.

        while(true)
        {
            Refill();
            if(Pop() != FORTH_TRUE) return;
            Interpret();
        }
    }

Then comes the word interpret. It tries to interpret the text between spaces first as a word, then as a number

    void Interpret()
    {
        while(true)
        {
            Bl();
            Word(inKeyword: true);
            if(IsEmptyWord()) { Drop(); break;};

            // TODO: remove string allocation from main loop. It is not trivial to do, because some functions inside InterpretWord rely on it.
            Dup();
            var aword = ToDotNetStringC().ToLowerInvariant();

            if(InterpretWord(aword)) continue;

            if(TryParseNumber(aword, out Cell n))
                InterpretNumber(n);
            else
                Throw($"{aword} is not a recognized word or number.");
        }
    }

## Interpreting numbers Let’s tackle parsing the number first. In Forth you can express numbers in different basis. We use the standard .NET conversion functions here, but those support just a few basis and throw exceptions on failure (bad design). We could consider writing our own to support all basis. Also the code can be compiled for 32 bits cell size and maybe it works.

    bool TryParseNumber(string s, out Cell n)
    {
        var b = ds[basep];

        try {
#if CELL32
            n = Convert.ToInt32(s, b);
#else
            n = Convert.ToInt64(s, b);
#endif
            return true;
        } catch (FormatException )
        {
            // This is not actually an exception case in Forth.
            n = 0;    

        } catch(OverflowException)
        {
            throw;
        }
        return false;
    }

Now that we can parse a number, we can interpret it. This is a token interpreter. Word as described as tokens of various kind. You are either executing a token, or compiling it inside of another word (i.e., when using the defining word :).

    void InterpretNumber(Cell value)
    {
        if(Executing)
            Execute(Token.Numb, value);
        else
            PushOp(Token.Numb, value);
    }

Executing a token is a bit complicated. Let’s look first at compiling it. Compilation in this model means adding a token on top of the data stack. In Forth, the data stack is where user defined words live and we are in the middle of defining one.

    void PushOp(Token op) {
        ds[herep] = (Code)op;
        herep++;
    }

Sometimes tokens have parameters. If they have a numeric one, we encode it so to minimize the overall size both in memory (for cache sake) and on disk. For that we use the GitVarInt library.

    void PushOp(Token op, Cell value)
    {
        PushOp(op);
        Write7BitEncodedCell(ds, herep, value, out var howMany);
        herep += howMany;
    }

## Interpreting words

Now that we know how to compile numbers and that it exists a magic function that executes tokens, we can go back and look at interpreting words. In this implementation, words are separated in user defined words, primitives and immediate primitive. This is likely a bad design brought about by a desire of optimize prematurely. There should be an unified representation.

The logic is still relatively simple. If it is a user defined word, execute/compile a call token with its address. If it is a primitive, execute/compile the corresponding token. If it is an immediate primitive, execute the code associated with it.

It is awkward that finding the user defined word relies on parameters on the stack, while the other cases use a .net string representation to find the token in an hash table. Apart from style, this is an irritating allocation in the main loop that could be optimized away with some work.

    bool InterpretWord(string aword)
    {
        LowerCase();
        FindUserDefinedWord();

        var found   = Pop();
        var xt      = (Index)Pop();

        // Manage user defined word.
        if(found != FORTH_FALSE)
        {
            var immediate = found == 1; // There can be user defined immediate functions.
            if(Executing || immediate)
                Execute(Token.Call, xt);
            else
                PushOp(Token.Call, xt);
            return true;
        }
        // Manage simple primitives.
        if(WordToSimpleOp.TryGetValue(aword, out var op))
        {
            if(Executing)
                Execute(op, null);
            else
                PushOp(op);
            return true;
        }
        // Manage immediate primitives.
        if(ImmediatePrimitives.TryGetValue(aword, out var immediateWord))
        {
            immediateWord.Item2();
            return true;
        }
        return false;
    }

Simple primitives map a word with the token defining it.

    readonly Dictionary<string, Token> WordToSimpleOp = new()
    {
        { "."           , Token.Prin },
        { "count"       , Token.Count },
        { "words"       , Token.Words },
        { "testsys"     , Token.TestSys },
        { "cells"       , Token.Cells },
        { "allot"       , Token.Allot },
        { "and"         , Token.And },
        { "or"          , Token.Or },
        { "base"        , Token.Base },
        { "refill"      , Token.Refill },
        { "interpret"   , Token.Interpret },
        { "quit"        , Token.Quit },
        { "word"        , Token.Word },
        { "parse"       , Token.Parse },
        { "save"        , Token.Save },
        { "load"        , Token.Load },
        { "savesys"     , Token.SaveSys },
        { "loadsys"     , Token.LoadSys },
        { "included"    , Token.Included },
        { ","           , Token.Comma },
        { "c,"          , Token.CComma },
        { "here"        , Token.Here },
        { "@"           , Token.At },
        { "c@"          , Token.CAt },
        { "pad"         , Token.Pad },
        { "!"           , Token.Store },
        { "state"       , Token.State },
        { "bl"          , Token.Bl },
        { ":"           , Token.Colo },
        { "bye"         , Token.Bye },
        { ".s"          , Token.DotS },
        { "+"           , Token.Plus },
        { "-"           , Token.Minu },
        { "*"           , Token.Mult },
        { "/"           , Token.Divi },
        { "<"           , Token.Less },
        { ">"           , Token.More },
        { "="           , Token.Equal },
        { "<>"          , Token.NotEqual },
        { "create"      , Token.Create },
        { "does>"       , Token.Does },
        { ">body"       , Token.Body },
        { "rdepth"      , Token.RDepth },
        { "swap"        , Token.Swap },
        { "depth"       , Token.Depth },
        { "over"        , Token.Over },
        { "dup"         , Token.Dup },
        { "dup2"        , Token.Dup2 },
        { "drop"        , Token.Drop },
        { "drop2"       , Token.Drop2 },
        { "*/mod"       , Token.MulDivRem },
        { "invert"      , Token.Invert },
        { "exit"        , Token.Exit },
        { "i"           , Token.I },
        { "j"           , Token.J },
        { ">r"          , Token.ToR },
        { "r>"          , Token.FromR },
        { "leave"       , Token.Leave },
        { "immediate"   , Token.Immediate },
        { "source"      , Token.Source },
        { "type"        , Token.Type },
        { "emit"        , Token.Emit },
        { "cr"          , Token.Cr },
        { "char"        , Token.Char },
        { ">in"         , Token.In },
        { "find"        , Token.Find },
        { "execute"     , Token.Exec },
        { ".net>type"   , Token.DType },
        { ".net>method" , Token.DMethod },
        { ".net>call"   , Token.DCall },
    };

On the other hand, immediate actions need to be executed at compile time when their token is encountered. This is indexed by the string word, which is what we see in the interpreter. But later I discovered the need to index it by operator as well, so I bolted it on as a tuple. We’ll look in more depth at how they work later.

    readonly Dictionary<string, (Token, Action)> ImmediatePrimitives = new();

User defined words are stored in the data space as a linked list. Each word is in the format: -> Cell <-> One Byte <-> Bytes <-> Bytes | Next Link| Len Word | Word chars | Tokens | This function follow the links, staring at dictHead, until it finds the word on top of the parameter stack. As an optimization, the length field uses the higher bit to store if the word is an immediate one (save one byte, save the planet).

    internal void FindUserDefinedWord()
    {
        var caddr = (Index)Pop();
        var clen  = ds[caddr];
        var cspan = new Span<AChar>(ds, caddr + 1 * CHAR_SIZE, clen);

        var dp = dictHead;

        while(true)
        {
            if(dp == 0) break;

            var wordNameStart = dp + CELL_SIZE;
            var wordLenRaw    = ds[wordNameStart];
            var wordLen       = Utils.ResetHighBit(wordLenRaw); // Resets the high bit, so we get the real length.
            var wordSpan      = new Span<AChar>(ds, wordNameStart + 1 * CHAR_SIZE, wordLen);
            var found         = cspan.SequenceEqual(wordSpan);
            if(found)
            {
                Push(LinkToCode(dp, wordLen));                
                var isImmediate = Utils.HighBitValue(wordLenRaw) == 1;
                Push( isImmediate ? 1 : -1);
                return;
            }
            dp = (Index)ReadCell(ds, dp);
        }
        // Not found
        Push(caddr);
        Push(0);
    }

These functions abstract out the details of the word structure. At least that was the idea. In practice that knowledge has leaked in other parts of the code base.

    static Index LinkToCode(Index link, Index wordLen)
        // Addr + Link size + len size  + word chars
        => link + CELL_SIZE + CHAR_SIZE + CHAR_SIZE * wordLen;

    static Index LinkToLen(Index link) => link + CELL_SIZE;

## Memory organization As we touched on the internal memory organization, it might now be time to describe it in more detail. Firstly the data space, parameter stack and return stack are represented as arrays of bytes. Some notes: * In Forth, you can access the stacks as Cells or bytes. I choose the lower denominator to simplify things. * I didn’t use the .NET Stack class because it doesn’t let you easily index into it. * One could use unsafe and pointers to avoid checking boundaries at each access, but then you could use the library just from unsafe code.

    Index sp     = 0;                   // Top of parameter stack.
    Index rp     = 0;                   // Top for return stack.
    Index herep  = 0;                   // Top of data space.
    AUnit[] ps;                         // Parameter stack.
    AUnit[] rs;                         // Return stack.
    AUnit[] ds;                         // Data space.

Then come some pointers that map areas in the data space that are used by the system or point to some system cells. They get initialized in the Vm constructor. Also, some other random state used by the system.

    Index dictHead;                     // The last word added to the dictionary.

    readonly Index source;              // Input buffer.
    readonly Index inp;                 // Points char number (0...source_max_chars) to be read in the input buffer.
    readonly Index inputBufferSize;     // Size of the input buffer.
    readonly Index keyWord;             // Word read by the interpreter.
    readonly Index word;                // Text read by the Forth word `word`. It needs to be separated from keyWord, so as to not conflict.
    readonly Index wordBufferSize;      // Size of the `word` and `keyword` buffer.
    readonly Index pad;                 // Pad area. A temporary area usable by the user.
    readonly Index dotnetStrings;       // Area used to store string parameters passed from dotnet to Forth. 
    readonly Index code;                // Each token needs to be stored in the data space before execution, so that the instruction pointer can work.
    readonly Index basep;               // Store base for numbers.
    readonly Index state;               // Are we compiling or interpreting? See `Executing` below.
    readonly Index userStart;           // Start of the user part of the data space. `Save` starts saving from here.
    readonly Index savedDictHead;       // Used by `save` and `load` to fetch the index of the last word in the dictionary.

    Index input_len_chars = 0;          // How many chars in the input buffer.

    bool Executing { get => ReadCell (ds, state) == FORTH_FALSE;
                     set => WriteCell(ds, state, value ? FORTH_FALSE : FORTH_TRUE);}

    public bool Debug { get ; set; }    // Doesn't do much as of now, but we can extend it to print out a lot of useful thins (i.e., token disassembly).

I decided for the system to use UTF8 internally to save space, despite .NET running on a ‘kind of’ UTF16. I convert at the boundary. A cell can be either 64 bits or 32 bits. I have not tested the latter.

    internal const Index CHAR_SIZE = 1;
    internal const Index CELL_SIZE = sizeof(Cell);

    const Cell FORTH_TRUE  = -1;
    const Cell FORTH_FALSE = 0;

This is very important. By setting this field you can get the interpret to read the next line of text from wherever (typically Console or file).

    public Func<string>? NextLine = null;

These cache the last dotnet type and method, so that you can call multiple methods on the same type ergonomically.

    Type lastType = typeof(Console);
    MethodInfo? lastMethod;

Let’s see how the pointers are initialized. First some debatable defaults for the most important data source areas.

    public Vm(
        Index parameterStackSize =  16 * 1_024,
        Index returnStackSize    =  16 * 1_024,
        Index dataStackSize      = 256 * 1_024,
        Index padSize            =       1_024,
        Index sourceSize         =       1_024,
        Index wordSize           =       1_024
        ) {

These are the arrays storing the three most important memory areas.

        ps   = new AUnit[parameterStackSize];
        rs   = new AUnit[returnStackSize];
        ds   = new AUnit[dataStackSize];

And we initialize all the pointers

        inputBufferSize  = sourceSize;
        wordBufferSize   = wordSize;

        code             = herep;
        herep           += CHAR_SIZE + CELL_SIZE; // Maximum size of an instruction.

        keyWord          = herep;
        herep           += wordSize * CHAR_SIZE;
        source           = herep;
        herep           += sourceSize * CHAR_SIZE;

        word             = herep;
        herep           += wordSize * CHAR_SIZE;

        pad              = herep;
        herep           += padSize;

        basep            = herep;
        herep           += CELL_SIZE;
        ds[basep]        = 10;

        dotnetStrings    = herep;
        herep           += 256 * Vm.CHAR_SIZE;

        inp              = herep;
        herep           += CELL_SIZE;

        state            = herep;
        herep           += CELL_SIZE;

        userStart        = herep;
        savedDictHead    = herep;
        herep           += CELL_SIZE;

This table contains the immediate primitive words. These words are executed at compile time. The management of conditionals and loop constructs is messy.

Take the if statement, at the point where the interpret encounters it, it doesn’t yet know where it has to jump to in case the condition is not satisfied. It embeds a ‘conditional’ branching instruction, leaving two bytes empty for the branching target. It also pushes the address of the empty bytes. When the interpret encounters else or then, it backfill those empty bytes with the right value so that the branch instruction jumps to the correct point.

Other flow control structures behave similarly.

        ImmediatePrimitives = new()
        {
            { "debug",      (Token.IDebug,     () => Debug = !Debug) },
            { "[char]",     (Token.IChar,      () => { Char(); PushOp(Token.Numb, Pop());}) },
            { "literal",    (Token.ILiteral,   () => { PushOp(Token.Numb, Pop());}) },
            { "sliteral",   (Token.ISLit,      EmbedSString(Token.ISLit)) },
            { "[",          (Token.IBrakO,     () => Executing = true) },
            { "]",          (Token.IBrakC,     () => Executing = false) },
            { ";",          (Token.ISemi,      () => { PushOp(Token.Exit);  Executing = true; }) },
            { "postpone",   (Token.IPostCall,  Postpone) },
            { "begin",      (Token.IBegin,     () => Push(herep)) },
            { "do",         (Token.IDo,        () => { PushOp(Token.Do); herep += CELL_SIZE; Push(herep); }) },
            { "loop",       (Token.ILoop,      EmbedHereJmp0Bck)},
            { "+loop",      (Token.ILoopP,     EmbedHereJmp0BckP)},
            { "again",      (Token.IAgain,     EmbedHereJmpBck)},    
            { "if",         (Token.IIf,        BranchAndMark) },
            { "else",       (Token.IElse,      EmbedInPoppedJmpFwd)  },    
            { "then",       (Token.IThen,      () => {
                var mark = (Index)Pop();
                short delta = (short)(herep - mark);
                WriteInt16(ds, mark, delta);
            }) },
            { "while",      (Token.IWhile,     BranchAndMark) },    
            { "repeat",     (Token.IRepeat,    () => {
                var whileMark = (Index)Pop();
                short delta   = (short)(herep + 3 - whileMark);
                WriteInt16(ds, whileMark, delta);
                EmbedHereJmpBck();
                }) },
            { "c\"",         (Token.ICStr,     EmbedString(Token.ICStr)) },
            { "s\"",         (Token.ISStr,     EmbedString(Token.ISStr)) },
        };

After everything is set up, we can now load the initialization files. In Forth, normally a good part of the interpret is written in Forth itself. Implementations vary with regard to how many instructions are primitives. The obvious trade-offs apply.

Here we try to load from a binary file first, if present.

So, that was the idea, at least. In practice I got tired of trying to figure out where such a file should live to be included correctly by the dll in all scenarios. So embedded as a constant instead. It also good so that you can just copy this single file in a project and be done.

        EvaluateMultipleLines(INIT_FORTH);
        /*
        if(File.Exists("init.io"))
        {
            FromDotNetString("init.io");
            LoadSystem(true);
        } else if(File.Exists("init.fth")) {
            FromDotNetString("init.fth");
            Included();
        } else
        {
            Console.WriteLine("No init file loaded.");
        }
        */

        Reset();
    }

In our token based interpret, loading and saving the system is a trivial operation. Just copy the bytes.

    void SaveSystem(bool all = false)
    {
        var start = all ? 0 : userStart ;
        WriteCell(ds, savedDictHead, dictHead);

        var fileName = ToDotNetString();
        File.WriteAllBytes(fileName, ds[start .. herep]);
        Console.WriteLine($"Saved in file {Path.Join(Environment.CurrentDirectory, fileName)} .");
    }
    void LoadSystem(bool all = false)
    {
        var start = all ? 0 : userStart ;
        var fileName = ToDotNetString();
        var buf      = File.ReadAllBytes(fileName);
        buf.CopyTo(ds, start);

        dictHead = (Index)ReadCell(ds, savedDictHead);
        herep = buf.Length;
        Console.WriteLine($"Loaded from file {Path.Join(Environment.CurrentDirectory, fileName)}");
    }

## Executing words

With all of that under our belt, we can now look at Execute. This is switch threaded. It could be written as a table of Token to Action, but then I would lose possible performance tricks that the compiler can play to speed it up (i.e., using function pointers).

It is a giant do ... while loop of getting the Token at the instruction pointer and performing the associated action. The action might change the instruction pointer (i.e., jumps), so when we get back to the top of the loop, we might be executing at a very different ip from where started.

We will comment on the most interesting cases.

    Index PushExecOp(Token op, Cell? value)
    {
        ds[code] = (Code)op;
        var howMany = 0;

        if(value is not null) Write7BitEncodedCell(ds, code + 1, (Cell)value, out howMany);
        return howMany + 1;
    }

    void Execute(Token op, Cell? data) {

        Index opLen = PushExecOp(op, data); // Store the token and data in the code area so IP can point to it.
        var ip      = code;

        do {
            var token = (Token)ds[ip];
            ip++;

            Cell n, flag, index, limit, incr ;
            Index count, idx;
            bool bflag;

            switch(token) {
                case Token.Words:
                    Console.WriteLine(string.Join(" ", Words()));
                    break;
                case Token.Source:
                    Source();
                    break;
                case Token.Base:
                    Push(basep);
                    break;
                case Token.Emit:
                    Console.Write((char)Pop());
                    break;
                case Token.Type:
                    Console.Write(ToDotNetString());
                    break;

A string literal gets stored immediately after the ip.

                case Token.ISLit:
                    count = ds[ip];
                    Push(ip + 1);
                    Push(count);
                    ip += count + 1;
                    break;

Postponing is a two step process. At compile time, we embed one of the two tokens below. They allow postponing either user defined functions or primitives. For immediate primitives, postponing means embedding a token to execute the action. See description of Postpone() later.

                case Token.IPostCall:
                    n = Read7BitEncodedCell(ds, ip, out count);
                    PushOp(Token.Call, n);
                    ip += count;
                    break;
                case Token.IPostponeOp:
                    PushOp((Token)ds[ip]);
                    ip += 2; // There is a noop here to classify this as 2 bytes operation.
                    break;
                case Token.ImmCall:
                    var act = ImmediateAction((Token)ds[ip]);
                    if(act is null) Throw($"ImmCall with a non existing op: {(Token)ds[ip]}");
                    act();
                    ip++;
                    break;
                case Token.Allot:
                    herep += (Index)Pop();
                    break;

These three tokens enable my rudimentary (static methods) dotnet integration.

                case Token.DType:
                    var typeName = ToDotNetString();
                    var aType = Type.GetType(typeName);
                    if(aType is null) Throw($"Cannot find type {typeName}");
                    lastType = aType;
                    break;
                case Token.DMethod:
                    var methodName = ToDotNetString();
                    lastMethod = lastType.GetMethod(methodName);
                    if(lastMethod is null) Throw($"No method {methodName} on type {lastType.Name}");
                    break;
                case Token.DCall:
                    DCall();
                    break;
                case Token.Cells:
                    Push(Pop() * CELL_SIZE);
                    break;
                case Token.Find:
                    Find();
                    break;
                case Token.Exec:
                    idx = (Index)Pop();
                    RPush(ip);
                    ip = (Index)idx;
                    break;
                case Token.Cr:
                    Console.WriteLine();
                    break;
                case Token.In:
                    Push(inp);
                    break;
                case Token.Char:
                    Char();
                    break;
                case Token.ICStr:
                    Push(ip);
                    ip += ds[ip] + 1;
                    break;
                case Token.ISStr:
                    Push(ip + 1);
                    idx = ds[ip];
                    Push(idx);
                    ip += idx + 1;
                    break;

Numb and NumbEx are two ways to embed a number in the ip stream. The latter is needed because of the control structures (i.e., loop, if, …). We need to use a full cell to store the jumping point as we don’t know upfront where we are going to jump to, hence how many bytes we are going to need to store the address.

                case Token.Numb:
                    n = Read7BitEncodedCell(ds, ip, out count);
                    Push(n);
                    ip += count;
                    break;
                case Token.NumbEx:
                    n = ReadCell(ds, ip);
                    Push(n);
                    ip = (Index)RPop();
                    break;
                case Token.Prin:
                    n = Pop();
                    Console.Write($"{Convert.ToString(n, ds[basep])} ");
                    break;
                case Token.Count:
                    Count();
                    break;
                case Token.Refill:
                    Refill();
                    break;
                case Token.Word:
                    Word();
                    break;
                case Token.Parse:
                    Parse();
                    break;
                case Token.Comma:
                    Comma();
                    break;
                case Token.CComma:
                    ds[herep] = (byte) Pop();
                    herep++;
                    break;
                case Token.Save:
                    SaveSystem();
                    break;
                case Token.Load:
                    LoadSystem();
                    break;
                case Token.SaveSys:
                    SaveSystem(true);
                    break;
                case Token.LoadSys:
                    LoadSystem(true);
                    break;
                case Token.Included:
                    Included();
                    break;
                case Token.Here:
                    Here();
                    break;
                case Token.ToR:
                    PStoRS();
                    break;
                case Token.FromR:
                    FromR();
                    break;
                case Token.At:
                    At();
                    break;
                case Token.CAt:
                    CFetch();
                    break;
                case Token.Pad:
                    Push(pad);
                    break;
                case Token.Drop:
                    Drop();
                    break;
                case Token.Drop2:
                    Drop2();
                    break;
                case Token.MulDivRem:
                    MulDivRem();
                    break;
                case Token.Store:
                    Store();
                    break;
                case Token.State:
                    State();
                    break;
                case Token.Bl:
                    Bl();
                    break;
                case Token.Dup:
                    Dup();
                    break;
                case Token.Swap:
                    Swap();
                    break;
                case Token.Over:
                    Over();
                    break;
                case Token.Dup2:
                    Dup2();
                    break;
                case Token.Bye:
                    Environment.Exit(0);
                    break;
                case Token.TestSys:
                    EvaluateMultipleLines(PRELIM_TEST);
                    break;
                case Token.DotS:
                    Console.WriteLine(StackString());
                    break;
                case Token.Quit:
                    Quit();
                    break;
                case Token.Interpret:
                    Interpret();
                    break;
                case Token.And:
                    Push(Pop() & Pop());
                    break;
                case Token.Or:
                    Push(Pop() | Pop());
                    break;
                case Token.Plus:
                    Push(Pop() + Pop());
                    break;
                case Token.Minu:
                    Push(- Pop() + Pop());
                    break;
                case Token.Mult:
                    Push(Pop() * Pop());
                    break;
                case Token.Less:
                    Push(Pop() > Pop() ? FORTH_TRUE : FORTH_FALSE);
                    break;
                case Token.More:
                    Push(Pop() < Pop() ? FORTH_TRUE : FORTH_FALSE);
                    break;
                case Token.Equal:
                    Push(Pop() == Pop() ? FORTH_TRUE : FORTH_FALSE);
                    break;
                case Token.NotEqual:
                    Push(Pop() != Pop() ? FORTH_TRUE : FORTH_FALSE);
                    break;
                case Token.Depth:
                    Push(sp / CELL_SIZE);
                    break;
                case Token.RDepth:
                    Push(rp / CELL_SIZE);
                    break;
                case Token.Invert:
                    Push(~Pop());
                    break;
                case Token.Immediate:
                    Utils.SetHighBit(ref ds[LinkToLen(dictHead)]);
                    break;
                case Token.Divi:
                    var d = Pop();
                    var u = Pop();
                    Push(u / d);
                    break;
                case Token.Body:
                    Push(Pop() + 1 + CELL_SIZE); // See create below.
                    break;

create ... does> is the pearl of Forth, but its implementation is complex. create gives a byte address in the data space a name adding it to the dictionary. It also embed a token in the instruction stream to push the created address when executed, so that this works create bob 10 , bob ?. Here is a place when we need to use NumbEx.

                case Token.Create:
                    Push(' ');
                    Word();
                    if(ds[Peek()] == 0) Throw("'create' needs a subsequent word in the stream.");
                    LowerCase();
                    DictAdd();

                    PushOp(Token.NumbEx);
                    // Need to use full cell because it gets substituted by a jmp in does>
                    // and I don't know how many cells the number I need to jump to takes
                    WriteCell(ds, herep, herep + CELL_SIZE);
                    herep += CELL_SIZE;
                    break;

Here stuff is tricky. First we get the address for the code of the last created word (where create stored the NumbEx addr). We then override it with a jump to the current ip. Then we store an instruction to push the address of the last created word. Finally, we copy all the remaining tokens until we encounter Exit. This allows the following to work and print 10: : var create , does> ? ; 10 var bob bob

                case Token.Does:
                    // Allows adding code to the last `defined` word, not just last `created` one!!
                    // Even in gforth!!
                    idx = Lastxt();
                    var addrToPush = ReadCell(ds, idx + 1);

                    var currentIp = herep;
                    herep = idx;
                    PushOp(Token.Jmp, currentIp);
                    herep = currentIp;
                    PushOp(Token.Numb, addrToPush);
                    PushUntilExit(ref ip);
                    ip = (Index)RPop();
                    break;

Colon :, despite its importance, is utterly simple. Get the next word from the input buffer, add it to the dictionary and move the interpret to compile mode.

                case Token.Colo:
                    Push(' ');
                    Word();
                    if(ds[Peek()] == 0) Throw("Colon needs a subsequent word in the stream.");
                    LowerCase();
                    DictAdd();
                    Executing = false;
                    break;

A call stores the ip on the return stack, while jmp just moves the ip.

                case Token.Call:
                    n = Read7BitEncodedCell(ds, ip, out count);
                    ip += count;
                    RPush(ip);
                    ip = (Index)n;
                    break;
                case Token.Jmp:
                    n = Read7BitEncodedCell(ds, ip, out _);
                    ip = (Index) n;
                    break;

exit gets the ip from the return stack

                case Token.Exit:
                    ip = (Index)RPop();
                    break;

Here come all the instructions for control flow control in a stack based Vm. I recommend stepping through a do ... loop to understand the tricky details. Generally, the return stack contains ip after the loop | limit | index. ‘ip after the loopis needed by theleave` instruction, which needs to know where to go. There might be a way to pre-calculate this at compile time, but I didn’t find it.

                case Token.Branch0:
                    flag = Pop();
                    ip += flag == FORTH_FALSE ? ReadInt16(ds, ip) : 2 ;
                    break;
                case Token.Do:
                    RPush(ReadCell(ds, ip));
                    ip += CELL_SIZE;
                    PStoRS();
                    PStoRS();
                    break;
                case Token.I:
                    Push(ReadCell(rs, rp - CELL_SIZE * 2));
                    break;
                case Token.J:
                    Push(ReadCell(rs, rp - CELL_SIZE * 5));
                    break;
                case Token.Leave:
                    RPop();
                    RPop();
                    ip = (Index)RPop();
                    break;
                case Token.Loop:
                    limit = RPop();
                    index = RPop();
                    index++;
                    bflag = index < limit; 
                    if(bflag) {
                        RPush(index);
                        RPush(limit);
                        ip += ReadInt16(ds, ip);
                    }  else {
                        RPop();
                        ip += 2;
                    }
                    break;
                case Token.LoopP:
                    incr  = Pop();
                    limit = RPop();
                    index = RPop();
                    index += incr;
                    bflag = incr > 0 ? index < limit : index >= limit; 
                    if(bflag) {
                        RPush(index);
                        RPush(limit);
                        ip += ReadInt16(ds, ip);
                    }  else {
                        RPop();
                        ip += 2;
                    }
                    break;

For control structures we use relative jumps of 2 bytes, not to waste a full cell for them. If you have an if statement longer than FFFFFFFF bytes, you are screwed.

                case Token.RelJmp:
                    ip += ReadInt16(ds, ip);
                    break;
                default:

We can treat all the immediate primitives in a generic way, by just calling their associated action.

                    var a = ImmediateAction(token); 
                    if(a is not null)
                        a();
                    else 
                        Throw($"{(Token)op} bytecode not supported.");
                    break;
            }

When do we stop looping? It took me a while to get this one right. The ip wonders around the data space following all the call and jmp,but, eventually, it comes back where it started.

        } while (ip != code + opLen);
    }

Some details Now you should understand the how interpretation and compilation work. The rest is details. Let’s look at a few of them. Forth uses included to interpret code in external files. Note the versatility of NextLine. We used it before as part of the interpret loop. Now we use the same mechanism to read from a file and keep track of line numbers for the sake of error messages.

    void Included()
    {

        var lineNum  = 0;
        var lineText = "";

        var fileName     = ToDotNetString();
        using var stream = File.OpenRead(fileName);
        using var reader = new StreamReader(stream);
        var backupNL     = NextLine;

        try {
            if (Debug) Console.Write($"Interpreting file {fileName} ...");
            NextLine = () => { lineNum++; lineText = reader.ReadLine()! ; return lineText; };

            Quit();

            if (Debug) Console.WriteLine(" done.\n");
        } catch(Exception)
        {
            ColorLine(ConsoleColor.Red, $"File: {fileName} Line: {lineNum}\n{lineText}");
            throw; 
        } finally
        {
            NextLine = backupNL;
        }
    }

This is awkward. When processing does>, I need to copy all the tokens until Exit. But how do I recognize it? There could be a byte with Exit value, which is part of some number. So I divide the tokens according to their lengths push each instruction one by one. There is certainly a better way, likely involving a more careful design of the token representation.

    void PushUntilExit(ref Index ip)
    {
        while(true)
        {
            var token = ds[ip];
            var count = 0;

            if(Utils.HasCellSize(token))
                count = CELL_SIZE;
            else if(Utils.HasVarNumberSize(token))
                Read7BitEncodedCell(ds, ip + 1, out count);
            else if(Utils.HasStringSize(token))
                count += ds[ip + 1] + 1; // Size of string + 1 for the len byte.

            Array.Copy(ds, ip, ds, herep, 1 + count);

            ip    += 1 + count;
            herep += 1 + count;

            if((Token)token == Token.Exit)
                break;
        }
    }

This long and ugly beast navigates the dictionary linked list until it finds a word. If it doesn’t, it looks for one in the primitives table, otherwise in the immediate primitive table.

    void Find()
    {
        // Look for user defined word
        var caddr = (Index)Peek();
        FindUserDefinedWord();

        if(Peek() != FORTH_FALSE) return;

        var clen  = ds[caddr];
        var cspan = new Span<AChar>(ds, caddr + 1 * CHAR_SIZE, clen);
        var sl    = Encoding.UTF8.GetString(cspan);

        // Look for simple statements
        if(WordToSimpleOp.TryGetValue(sl, out var op))
        {
            RetNewOp(op);
            return;
        }
        // Look for immediate words
        if(ImmediatePrimitives.TryGetValue(sl, out var imm))
        {
            RetNewImmediateOp(imm.Item1);
            return;
        }
        // Getting here, we return the result of FindUserDefinedWord. Below utility funcs.
        void RetNewImmediateOp(Token op)
        {
            var xt = herep;
            PushOp(Token.ImmCall);
            PushOp(op);
            PushOp(Token.Exit);
            Ret(xt, 1);
        }
        void RetNewOp(Token op)
        {
            var xt = herep;
            PushOp(op);
            PushOp(Token.Exit);
            Ret(xt, -1);
        }
        void Ret(Index xt, Cell f)
        {
            Drop(); Drop(); Push(xt); Push(f);
        }
    }
    Index Lastxt() => LinkToCode(dictHead, ds[dictHead + CELL_SIZE]);

Adding to the dictionary means adding to a linked list. Just bytes twiddling.

    internal void DictAdd()
    {
        // First put the link
        Push(dictHead);             // Push last index
        dictHead = herep;           // DH is now here
        Comma();                    // Store last index in new dictHead (here)

        // Copy word to here
        var len = ds[(Index)Peek()];
        Push(herep);
        Push(len + 1);
        CMove();
        herep += len + 1;
    }

Postpone gave me some grief. In summary, if it is a user defined word, postpone a call to its xt; if it is a primitive, postpone its token; if it is an immediate primitive, postpone its token.

    void Postpone()
    {
        Bl();
        Word();
        LowerCase();
        Dup();
        var sl = ToDotNetStringC();

        FindUserDefinedWord();

        var res = Pop();
        var xt    = (Index)Pop();

        if(res != FORTH_FALSE) {
            PushOp(Token.IPostCall, xt);
            return;
        }

        if(WordToSimpleOp.TryGetValue(sl, out var op)) {
            PushOp(Token.IPostponeOp);
            PushOp(op);
            PushOp(Token.Noop); // Need that because we don't have tokens with 2 bytes operands.
            return;
        }

        // TODO: confirm that all immediate tokens are one byte long.
        if(ImmediatePrimitives.TryGetValue(sl, out var imm)) {
            PushOp(imm.Item1);
            return;
        }
        Throw($"{sl: don't know this word.}");
    }

## List of Tokens In each implementation, each instruction is (Token, Operand) where Token is one byte, while operand can be 1 byts, 2 bytes, Cell size, a variable number or a string. There is vast literature, but no agreement on the most optimal number of primitives for a Forth system. It is a trade off between easy of porting vs performance and compactness. I stayed somewhere in the middle. Whatever seemed to be highly optimizable as a single instruction became one. Having said that, if I had to do it again, I would probably have less primitives.

    internal enum Token {
        Error , Colo,  Does, Plus, Minu, Mult, Divi, Prin, Base, Noop,
        Count, Word, Parse, Refill, Comma, CComma, Here, At, Store, State, Bl, Dup, Exit, Immediate,
        Swap, Dup2, Drop, Drop2, Find, Bye, DotS, Interpret, Quit, Create, Body, RDepth, Depth,
        Less, Words, TestSys, More, Equal, NotEqual, Do, Loop, LoopP, ToR, FromR, I, J, Leave, Cr,
        Source, Type, Emit, Char, In, Over, And, Or, Allot, Cells, Exec, Invert, MulDivRem,
        Save, Load, SaveSys, LoadSys, Included, DType, DCall, DMethod, CAt, Pad,
        IDebug, ISemi,  IBegin, IDo, ILoop, ILoopP, IAgain, IIf, IElse, IThen,
        IWhile, IRepeat, IBrakO, IBrakC,   // End of 1 byte
        Branch0, RelJmp, ImmCall, IPostponeOp,// End of 2 byte size
        NumbEx, // End of CELL Size 
        Jmp , Numb, Call, IPostCall, ILiteral, IChar,// End of Var number
        ICStr, ISStr, ISLit, // End of string words
        FirstHasVarNumb = Jmp, FirstHas2Size = Branch0, FirstHasCellSize = NumbEx,
        FirstStringWord = ICStr,
    }

## Forth defined primitives Whatever is not a primitive, is implemented in Forth in the giant string below. Note that comments are implemented in Forth. What other language lets you do that? I guess Lisp, SmallTalk and derivatives …

    const string INIT_FORTH = @"
: ( [char] ) parse drop drop ; immediate
: \ 0 word drop ; immediate

\ Some modified from https://theforth.net/package/minimal/current-view/README.md
: variable create 0 , ;
: constant create , does> @ ;

\ Arithmetic
: 1+ 1 + ;
: 2+ 2 + ;
: 1- 1 - ;
: 2- 2 - ;
: min ( n1 n2 -- n3 )  over over > if swap then drop ;
: max ( n1 n2 -- n3 )  over over < if swap then drop ;
: mod ( n n -- n )  1 swap */mod drop ;
: dec 10 base ! ;
: hex 16 base ! ;
: 2* 2 * ;
: negate -1 * ;
: d- - ;

\ Stack
: rot ( x1 x2 x3 -- x2 x3 x1 )  >r  swap r> swap ;
: -rot ( x1 x2 x3 -- x3 x2 x1 )  rot rot ;
: nip ( x1 x2 -- x2 )  swap drop ;
: tuck ( x1 x2 -- x2 x1 x2 )  swap over ;
: ?dup dup 0 <> if dup then ;
: bounds ( addr1 u -- addr2 addr3 )  over + swap ;
: 2dup ( d1 -- d1 d1 )  over over ;
: 2swap ( d1 d2 -- d2 d1 )  >r rot rot r> rot rot ;
: 2over ( d1 d2 -- d1 d2 d1 )  >r >r 2dup r> r> 2swap ;
: um/mod 2dup mod -rot / ;

\ Boolean
0 constant false
false invert constant true
: 0= 0 = ;
: 0< 0 < ;
: 0> 0 > ;
: or ( x x -- x )  invert swap invert and invert ; ( do morgan )
: xor ( x x -- x )  over over invert and >r swap invert and r> or ;
: lshift ( x1 u -- x2 )  begin dup while >r  2*  r> 1 - repeat drop ;
: endif postpone then ; immediate

\ Memory
: ? @ . ;
: +! ( x addr -- )  swap over @ + swap ! ;
: chars ;
: char+ ( c-addr1 -- c-addr2 )  1 chars + ;
: cell+ ( addr1 -- addr2 )  1 cells + ;
: aligned ( addr -- a-addr )  cell+ 1 -   1 cells 1 - invert  and ;
: 2! ( d addr -- )   SWAP OVER ! CELL+ ! ;
: 2@ ( addr -- d )  DUP CELL+ @ SWAP @ ;

\ Compiler
: ' bl word find drop ;
: ['] ' postpone literal ; immediate
: value ( -- )  create , does> @ ;
: defer ( ""<spaces>name"" -- )  create 0 , does> @ execute ;
: to ( x ""<spaces>name"" -- ) 
   state @ 
   if  postpone [']  postpone >body postpone !  
   else ' >body ! then ; immediate

: is ( x ""<spaces>name"" -- ) 
   state @ if  postpone to  else ['] to execute  then ; immediate

\ Strings
: space ( -- )  bl emit ;
: spaces ( u -- ) dup 0 > if  begin dup while  space 1 -  repeat  then  drop ;

\ .net inteop samples
: .net ( type-s-addr type-c methodName-s-addr method-name-c -- ** )
    2swap .net>type .net>method .net>call ;

: escape s"" System.Uri, System"" s"" EscapeDataString"" .net ;
: sqrt s"" System.Math"" s"" Sqrt"" .net ;
";

The rest is either byte fiddling or control structures details. You should be able to understand it on your own, given what explained before. If you are highly motivated.

    void CFetch()
    {
        var c = (Index)Pop();
        var sl = new Span<AUnit>(ds, c, 1);
        Push(sl[0]);
    }

    internal void Count()
    {
        var start = (Index) Pop();
        Push(start + 1);
        Push(ds[start]);
    }
    void FromDotNetString(string s)
    {
        var bytes = Encoding.UTF8.GetBytes(s);
        bytes.CopyTo(ds, dotnetStrings);
        Push(dotnetStrings);
        Push(bytes.Length);
    }
    internal string ToDotNetStringC()
    {
        var a = (Index)Pop();
        var s = new Span<AChar>(ds, a + CHAR_SIZE, ds[a]);
        return Encoding.UTF8.GetString(s);
    }
    internal string ToDotNetString()
    {
        var c = (Index)Pop();
        var a = (Index)Pop();
        var s = new Span<AUnit>(ds, a, c);
        return Encoding.UTF8.GetString(s);
    }
    internal void Parse()
    {
        var delim = (byte) Pop();
        ref var off = ref ds[inp]; 
        var addr = source + off;
        var startOff = off;

        while(ds[source + off] != delim) off++;
        off++;

        Push(addr);
        Push(off - startOff - 1);
    }
    /* TODO: the delimiter in this implementation (and Forth) as to be one byte char, but UTF8 puts that into question */
    internal void Word(bool inKeyword = false)
    {
        var delim = (byte)Pop();
        var s = ToChars(source, input_len_chars);
        var toPtr = inKeyword ? keyWord : word;

        var w = ToChars(toPtr, wordBufferSize);

        var j = 1; // It is a counted string, the first byte contains the length

        ref var index = ref ds[this.inp];

        while (index < input_len_chars && s[(Index)index] == delim) { index++; }

        // If all spaces to the end of the input, return a string with length 0. 
        if (index >= input_len_chars)
        {
            w[0] = (byte)0;
            Push(toPtr);
            return;
        }

        // Copy chars until end of space allocated, end of buffer or delim.
        while (j < wordBufferSize && index < input_len_chars && s[(Index)index] != delim)
        {
            var c = s[(Index)index];
            index++;
            w[j++] = c;
        }
        // Points past the delimiter. Otherwise it would stay on last " of a string.
        if(index < input_len_chars) index++;
        if (j >= wordBufferSize) throw new Exception($"Word longer than {wordBufferSize}: {Encoding.UTF8.GetString(s)}");

        w[0] = (byte)(j - 1);  // len goes into the first char
        Push(toPtr);
    }
    Span<byte> ToChars(Index start, Index lenInBytes)
            => new(ds, start, lenInBytes);
    const string PRELIM_TEST = @"
CR CR SOURCE TYPE ( Preliminary test ) CR
SOURCE ( These lines test SOURCE, TYPE, CR and parenthetic comments ) TYPE CR
( The next line of output should be blank to test CR ) SOURCE TYPE CR CR

( It is now assumed that SOURCE, TYPE, CR and comments work. SOURCE and      )
( TYPE will be used to report test passes until something better can be      )
( defined to report errors. Until then reporting failures will depend on the )
( system under test and will usually be via reporting an unrecognised word   )
( or possibly the system crashing. Tests will be numbered by #n from now on  )
( to assist fault finding. Test successes will be indicated by               )
( 'Pass: #n ...' and failures by 'Error: #n ...'                             )

( Initial tests of >IN +! and 1+ )
( Check that n >IN +! acts as an interpretive IF, where n >= 0 )
( Pass #1: testing 0 >IN +! ) 0 >IN +! SOURCE TYPE CR
( Pass #2: testing 1 >IN +! ) 1 >IN +! xSOURCE TYPE CR
( Pass #3: testing 1+ ) 1 1+ >IN +! xxSOURCE TYPE CR

( Test results can now be reported using the >IN +! trick to skip )
( 1 or more characters )

( The value of BASE is unknown so it is not safe to use digits > 1, therefore )
( it will be set it to binary and then decimal, this also tests @ and ! )

( Pass #4: testing @ ! BASE ) 0 1+ 1+ BASE ! BASE @ >IN +! xxSOURCE TYPE CR
( Set BASE to decimal ) 1010 BASE !
( Pass #5: testing decimal BASE ) BASE @ >IN +! xxxxxxxxxxSOURCE TYPE CR

( Now in decimal mode and digits >1 can be used )

( A better error reporting word is needed, much like .( which can't  )
( be used as it is in the Core Extension word set, similarly PARSE can't be )
( used either, only WORD is available to parse a message and must be used   )
( in a colon definition. Therefore a simple colon definition is tested next )

( Pass #6: testing : ; ) : .SRC SOURCE TYPE CR ; 6 >IN +! xxxxxx.SRC
( Pass #7: testing number input ) 19 >IN +! xxxxxxxxxxxxxxxxxxx.SRC

( VARIABLE is now tested as one will be used instead of DROP e.g. Y ! )

( Pass #8: testing VARIABLE ) VARIABLE Y 2 Y ! Y @ >IN +! xx.SRC

: MSG 41 WORD COUNT ;  ( 41 is the ASCII code for right parenthesis )
( The next tests MSG leaves 2 items on the data stack )
( Pass #9: testing WORD COUNT ) 5 MSG abcdef) Y ! Y ! >IN +! xxxxx.SRC
( Pass #10: testing WORD COUNT ) MSG ab) >IN +! xxY ! .SRC

( For reporting success .MSG( is now defined )
: .MSG( MSG TYPE ; .MSG( Pass #11: testing WORD COUNT .MSG) CR

( To define an error reporting word, = 2* AND will be needed, test them first )
( This assumes 2's complement arithmetic )
1 1 = 1+ 1+ >IN +! x.MSG( Pass #12: testing = returns all 1's for true) CR
1 0 = 1+ >IN +! x.MSG( Pass #13: testing = returns 0 for false) CR
1 1 = -1 = 1+ 1+ >IN +! x.MSG( Pass #14: testing -1 interpreted correctly) CR

1 2* >IN +! xx.MSG( Pass #15: testing 2*) CR
-1 2* 1+ 1+ 1+ >IN +! x.MSG( Pass #16: testing 2*) CR

-1 -1 AND 1+ 1+ >IN +! x.MSG( Pass #17: testing AND) CR
-1  0 AND 1+ >IN +! x.MSG( Pass #18: testing AND) CR
6  -1 AND >IN +! xxxxxx.MSG( Pass #19: testing AND) CR

( Define ~ to use as a 'to end of line' comment. \ cannot be used as it a )
( Core Extension word )
: ~  ( -- )  SOURCE >IN ! Y ! ;

( Rather than relying on a pass message test words can now be defined to )
( report errors in the event of a failure. For convenience words ?T~ and )
( ?F~ are defined together with a helper ?~~ to test for TRUE and FALSE  )
( Usage is: <test> ?T~ Error #n: <message>                               )
( Success makes >IN index the ~ in ?T~ or ?F~ to skip the error message. )
( Hence it is essential there is only 1 space between ?T~ and Error      )

: ?~~  ( -1 | 0 -- )  2* >IN +! ;
: ?F~ ( f -- )   0 = ?~~ ;
: ?T~ ( f -- )  -1 = ?~~ ;

( Errors will be counted )
VARIABLE #ERRS 0 #ERRS !
: Error  1 #ERRS +! -6 >IN +! .MSG( CR ;
: Pass  -1 #ERRS +! 1 >IN +! Error ;  ~ Pass is defined solely to test Error

-1 ?F~ Pass #20: testing ?F~ ?~~ Pass Error
-1 ?T~ Error #1: testing ?T~ ?~~ ~

0  0 = 0= ?F~ Error #2: testing 0=
1  0 = 0= ?T~ Error #3: testing 0=
-1 0 = 0= ?T~ Error #4: testing 0=

0  0 = ?T~ Error #5: testing =
0  1 = ?F~ Error #6: testing =
1  0 = ?F~ Error #7: testing =
-1 1 = ?F~ Error #8: testing =
1 -1 = ?F~ Error #9: testing =

-1 0< ?T~ Error #10: testing 0<
0  0< ?F~ Error #11: testing 0<
1  0< ?F~ Error #12: testing 0<

 DEPTH 1+ DEPTH = ?~~ Error #13: testing DEPTH
 ~ Up to now whether the data stack was empty or not hasn't mattered as
 ~ long as it didn't overflow. Now it will be emptied - also
 ~ removing any unreported underflow
 DEPTH 0< 0= 1+ >IN +! ~ 0 0 >IN ! Remove any underflow
 DEPTH 0= 1+ >IN +! ~ Y !  0 >IN ! Empty the stack
 DEPTH 0= ?T~ Error #14: data stack not emptied 

 4 -5 SWAP 4 = SWAP -5 = = ?T~ Error #15: testing SWAP
 111 222 333 444
 DEPTH 4 = ?T~ Error #16: testing DEPTH
 444 = SWAP 333 = = DEPTH 3 = = ?T~ Error #17: testing SWAP DEPTH
 222 = SWAP 111 = = DEPTH 1 = = ?T~ Error #18: testing SWAP DEPTH
 DEPTH 0= ?T~ Error #19: testing DEPTH = 0

~ From now on the stack is expected to be empty after a test so
~ ?~ will be defined to include a check on the stack depth. Note
~ that ?~~ was defined and used earlier instead of ?~ to avoid
~ (irritating) redefinition messages that many systems display had
~ ?~ simply been redefined

: ?~  ( -1 | 0 -- )  DEPTH 1 = AND ?~~ ; ~ -1 test success, 0 test failure

123 -1 ?~ Pass #21: testing ?~
Y !   ~ equivalent to DROP

~ Testing the remaining Core words used in the Hayes tester, with the above
~ definitions these are straightforward

1 DROP DEPTH 0= ?~ Error #20: testing DROP
123 DUP  = ?~ Error #21: testing DUP
123 ?DUP = ?~ Error #22: testing ?DUP
0  ?DUP 0= ?~ Error #23: testing ?DUP
123  111  + 234  = ?~ Error #24: testing +
123  -111 + 12   = ?~ Error #25: testing +
-123 111  + -12  = ?~ Error #26: testing +
-123 -111 + -234 = ?~ Error #27: testing +
-1 NEGATE 1 = ?~ Error #28: testing NEGATE
0  NEGATE 0=  ?~ Error #29: testing NEGATE
987 NEGATE -987 = ?~ Error #30: testing NEGATE
HERE DEPTH SWAP DROP 1 = ?~ Error #31: testing HERE
CREATE TST1 HERE TST1 = ?~ Error #32: testing CREATE HERE
16  ALLOT HERE TST1 NEGATE + 16 = ?~ Error #33: testing ALLOT
-16 ALLOT HERE TST1 = ?~ Error #34: testing ALLOT
0 CELLS 0= ?~ Error #35: testing CELLS
1 CELLS ALLOT HERE TST1 NEGATE + VARIABLE CSZ CSZ !
CSZ @ 0= 0= ?~ Error #36: testing CELLS
3 CELLS CSZ @ DUP 2* + = ?~ Error #37: testing CELLS
-3 CELLS CSZ @ DUP 2* + + 0= ?~ Error #38: testing CELLS
: TST2  ( f -- n )  DUP IF 1+ THEN ;
0 TST2 0=  ?~ Error #39: testing IF THEN
1 TST2 2 = ?~ Error #40: testing IF THEN
: TST3  ( n1 -- n2 )  IF 123 ELSE 234 THEN ;
0 TST3 234 = ?~ Error #41: testing IF ELSE THEN
1 TST3 123 = ?~ Error #42: testing IF ELSE THEN
: TST4  ( -- n )  0 5 0 DO 1+ LOOP ;
TST4 5 = ?~ Error #43: testing DO LOOP
: TST5  ( -- n )  0 10 0 DO I + LOOP ;
TST5 45 = ?~ Error #44: testing I
: TST6  ( -- n )  0 10 0 DO DUP 5 = IF LEAVE ELSE 1+ THEN LOOP ;
TST6 5 = ?~ Error #45: testing LEAVE
: TST7  ( -- n1 n2 ) 123 >R 234 R> ;
TST7 NEGATE + 111 = ?~ Error #46: testing >R R>
: TST8  ( -- ch )  [CHAR] A ;
TST8 65 = ?~ Error #47: testing [CHAR]
: TST9  ( -- )  [CHAR] s [CHAR] s [CHAR] a [CHAR] P 4 0 DO EMIT LOOP ;
TST9 .MSG(  #22: testing EMIT) CR
: TST10  ( -- )  S"" Pass #23: testing S"" TYPE [CHAR] "" EMIT CR ; TST10

~ The Hayes core test core.fr uses CONSTANT before it is tested therefore
~ we test CONSTANT here

1234 CONSTANT CTEST
CTEST 1234 = ?~ Error #48: testing CONSTANT

~ The Hayes tester uses some words from the Core extension word set
~ These will be conditionally defined following definition of a
~ word called ?DEFINED to determine whether these are already defined

VARIABLE TIMM1 0 TIMM1 !
: TIMM2  123 TIMM1 ! ; IMMEDIATE
: TIMM3 TIMM2 ; TIMM1 @ 123 = ?~ Error #49: testing IMMEDIATE

: ?DEFINED  ( ""name"" -- 0 | -1 )  32 WORD FIND SWAP DROP 0= 0= ;
?DEFINED SWAP 0= ?~ Error #50: testing FIND ?DEFINED
?DEFINED <<no-such-word-hopefully>> 0= ?~ Error #51 testing FIND ?DEFINED

?DEFINED \ ?~ : \ ~ ; IMMEDIATE 
\ Error #52: testing \
: TIMM4  \ Error #53: testing \ is IMMEDIATE
;

~ TRUE and FALSE are defined as colon definitions as they have been used
~ more than CONSTANT above

?DEFINED TRUE  ?~ : TRUE 1 NEGATE ;
?DEFINED FALSE ?~ : FALSE 0 ;
?DEFINED HEX   ?~ : HEX 16 BASE ! ;

TRUE -1 = ?~ Error #54: testing TRUE
FALSE 0=  ?~ Error #55: testing FALSE
10 HEX 0A = ?~ Error #56: testing HEX
AB 0A BASE ! 171 = ?~ Error #57: testing hex number

~ Delete the ~ on the next 2 lines to check the final error report
~ Error #998: testing a deliberate failure
~ Error #999: testing a deliberate failure

~ Describe the messages that should be seen. The previously defined .MSG(
~ can be used for text messages

CR .MSG( Results: ) CR
CR .MSG( Pass messages #1 to #23 should be displayed above)
CR .MSG( and no error messages) CR

~ Finally display a message giving the number of tests that failed.
~ This is complicated by the fact that untested words including .( ."" and .
~ cannot be used. Also more colon definitions shouldn't be defined than are
~ needed. To display a number, note that the number of errors will have
~ one or two digits at most and an interpretive loop can be used to
~ display those.

CR
0 #ERRS @
~ Loop to calculate the 10's digit (if any)
DUP NEGATE 9 + 0< NEGATE >IN +! ( -10 + SWAP 1+ SWAP 0 >IN ! )
~ Display the error count
SWAP ?DUP 0= 1+ >IN +! ( 48 + EMIT ( ) 48 + EMIT

.MSG(  test) #ERRS @ 1 = 1+ >IN +! ~ .MSG( s)
.MSG(  failed out of 57 additional tests) CR

CR CR .MSG( --- End of Preliminary Tests --- ) CR

";
}

public class ForthException: Exception { public ForthException(string s): base(s) { } };

static class Utils {

    const AChar HighBit     = 0b10000000; 
    const AChar HighBitMask = 0b01111111;

    internal static bool  IsHighBitSet(AChar c)     => (HighBit & c) != 0;
    internal static AChar ResetHighBit(AChar c)     => (AChar)(HighBitMask & c);
    internal static void SetHighBit(ref AChar c)    => c = (AChar)(HighBit | c);
    internal static AChar HighBitValue(AChar c)     => (AChar) (c >> 7);

    internal static void WriteInt16(AUnit[] ar, Index i, Int16 c)
        => BinaryPrimitives.WriteInt16LittleEndian(new Span<byte>(ar, i, 2), c);
    internal static Int16 ReadInt16(AUnit[] ar, Index i)
        => BinaryPrimitives.ReadInt16LittleEndian(new Span<byte>(ar, i, 2));

    internal static void WriteCell(AUnit[] ar, Index i, Cell c)
#if CELL32
        => BinaryPrimitives.WriteInt32LittleEndian(new Span<byte>(ar, i, Vm.CELL_SIZE), c);
#else
        => BinaryPrimitives.WriteInt64LittleEndian(new Span<byte>(ar, i, Vm.CELL_SIZE), c);
#endif

    internal static Cell ReadCell(AUnit[] ar, Index i)
#if CELL32
        => BinaryPrimitives.ReadInt32LittleEndian(new Span<byte>(ar, i, Vm.CELL_SIZE));
#else
        => BinaryPrimitives.ReadInt64LittleEndian(new Span<byte>(ar, i, Vm.CELL_SIZE));
#endif

    internal static AUnit[] Add(AUnit[] a, ref Index i, Cell t) {
        if(i + Vm.CELL_SIZE >= a.Length) Array.Resize(ref a, i * 2);
        WriteCell(a, i, t);        
        i += Vm.CELL_SIZE;
        return a;
    }
    internal static Cell ReadBeforeIndex(AUnit[] a, ref Index i)
    {
        i -= Vm.CELL_SIZE;
        return ReadCell(a, i);
    }
    [DoesNotReturn]
    internal static void Throw(string message) => throw new ForthException(message);

    internal static long Read7BitEncodedCell(Code[] codes, Index index, out Index howMany) {
        using var stream = new MemoryStream(codes, index, 10);
        howMany = (Index)stream.Position;
#if CELL32
        var result = stream.ReadVarInt32();
#else
        var result = stream.ReadVarInt64();
#endif
        howMany = (Index)stream.Position - howMany;
        return result;
    }

    internal static void Write7BitEncodedCell(Code[] codes, Index index, Cell value, out Index howMany) {
        //howMany = Write(codes, index, value);
        using var stream = new MemoryStream(codes, index, 10);
        howMany = (Index)stream.Position;
        stream.WriteVarInt(value);
        howMany = (Index)stream.Position - howMany;
    }

    internal static bool Has2NumberSize(byte b)
        => b >= (int)Vm.Token.FirstHas2Size && b < (int)Vm.Token.FirstHasCellSize;
    internal static bool HasCellSize(byte b)
        => b >= (int)Vm.Token.FirstHasCellSize && b < (int)Vm.Token.FirstHasVarNumb;
    internal static bool HasVarNumberSize(byte b)
        => b >= (int)Vm.Token.FirstHasVarNumb && b < (int) Vm.Token.FirstStringWord;
    internal static bool HasStringSize(byte b)
        => b >= (int)Vm.Token.FirstStringWord;

    // Rewritten from https://github.com/pvginkel/GitVarInt/blob/master/GitVarInt/VarInt.cs
    // to remove unsafe code and use new C# feature with the goal of removing any dependencies and be able to copy the file to a project.
    static int ReadVarInt32(this Stream stream) => ZigZagDecode(stream.ReadVarUInt32());

    static uint ReadVarUInt32(this Stream stream)
    {
        byte b = ReadByte(stream);
        uint value = (uint)(b & 127);

        while ((b & 128) != 0)
        {
            value += 1;
            if (value == 0 || (value & 0xfe000000u) != 0)
                Throw("Decode overflow");
            b = ReadByte(stream);
            value = (value << 7) + (uint)(b & 127);
        }

        return value;
    }

    static long ReadVarInt64(this Stream stream) => ZigZagDecode(stream.ReadVarUInt64());

    static ulong ReadVarUInt64(this Stream stream)
    {
        byte b = ReadByte(stream);
        ulong value = (ulong)(b & 127);

        while ((b & 128) != 0)
        {
            value += 1;
            if (value == 0 || (value & 0xfe00000000000000ul) != 0)
                Throw("Decode overflow");
            b = ReadByte(stream);
            value = (value << 7) + (ulong)(b & 127);
        }

        return value;
    }

    static void WriteVarInt(this Stream stream, int value) => stream.WriteVarInt(ZigZagEncode(value));

    static void WriteVarInt(this Stream stream, uint value)
    {
        Span<byte> buffer = stackalloc byte[5];
        int offset = 5;

        buffer[--offset] = (byte)(value & 0x7F);

        while ((value >>= 7) != 0)
            buffer[--offset] = (byte)(0x80 | (--value & 0x7F));

        while (offset < 5)
            stream.WriteByte(buffer[offset++]);
    }

    static void WriteVarInt(this Stream stream, long value) => stream.WriteVarInt(ZigZagEncode(value));

    static void WriteVarInt(this Stream stream, ulong value)
    {
        Span<byte> buffer = stackalloc byte[10];
        int offset = 10;

        buffer[--offset] = (byte)(value & 0x7F);

        while ((value >>= 7) != 0)
            buffer[--offset] = (byte)(0x80 | (--value & 0x7F));

        while (offset < 10)
            stream.WriteByte(buffer[offset++]);
    }

    static uint  ZigZagEncode(int value)   => ((uint) value << 1) ^ (uint)-(int)((uint)value >> 31);
    static ulong ZigZagEncode(long value)  => ((ulong)value << 1) ^ (ulong)-(long)((ulong)value >> 63);
    static int   ZigZagDecode(uint value)  => (int)  (value >> 1) ^ -((int)value & 0x1);
    static long  ZigZagDecode(ulong value) => (long) (value >> 1) ^ -((long)value & 0x1);

    private static byte ReadByte(Stream stream)
    {
        int b = stream.ReadByte();
        if (b == -1)
            Throw("Unexpected end");

        return (byte)b;
    }
}

System.Char[]
    void CMove()
    {
        var u  = Pop();
        var a2 = Pop();
        var a1 = Pop();
        Array.Copy(ds, a1, ds, a2, u);
    }
    void BranchAndMark() { PushOp(Token.Branch0); Push(herep); herep += 2;}
    void EmbedInPoppedJmpFwd() {
            PushOp(Token.RelJmp);
            var mark = (Index)Pop();
            short delta = (short)((herep + 2) - mark);
            WriteInt16(ds, mark, delta);
            Push(herep);
            herep += 2;
    }
    void EmbedHereJmpBck() {
            PushOp(Token.RelJmp);
            var mark = (Index)Pop();
            var delta = (short)(mark - herep);
            WriteInt16(ds, herep, delta); 
            herep += 2;
    }
    void EmbedHereJmp0Bck() {
            PushOp(Token.Loop);
            var mark = (Index)Pop();
            var delta = (short)(mark - herep);
            WriteInt16(ds, herep, delta); 
            herep += 2;

            var leaveTarget = mark - CELL_SIZE;
            WriteCell(ds, leaveTarget, herep);
    }
    void EmbedHereJmp0BckP() {
            PushOp(Token.LoopP);
            var mark = (Index)Pop();
            var delta = (short)(mark - herep);
            WriteInt16(ds, herep, delta); 
            herep += 2;
    }
    Action EmbedString(Token op) { return () =>
    {
        PushOp(op);

        Push('"');
        Parse();
        var len = (Index)Peek();
        Push(herep + 1);
        Swap();
        CMove();
        if(Executing && op == Token.ICStr) Push(herep);
        if(Executing && op == Token.ISStr) { Push(herep + 1); Push(len); }
        ds[herep] = (byte)len;
        herep += len + 1;
    }; }

    Action EmbedSString(Token op) { return () =>
    {
        PushOp(op);
        var len = (Index)Pop();
        Push(herep + 1);
        Push(len);
        CMove();
        ds[herep] = (byte)len;
        herep += len + 1;
    };}
    public void Reset()
    {
        sp = 0; rp = 0; Executing = true; ds[inp] = 0;
    }
    public void EvaluateSingleLine(string forthLine)
    {
        var oldLine = NextLine;
        try { 
            NextLine = () => forthLine;
            Refill();
            if(Pop() == FORTH_TRUE)
                Interpret();
        } finally
        {
            NextLine = oldLine;
        }
    }
    // Code duplication from EvaluateSingleLine on purpose for perf reason. TODO: Refactor EvaluateMultipleLines.
    public void EvaluateMultipleLines(string forthCode)
    {
        var oldLine = NextLine;
        try {
            var s = "";
            var forthLines = forthCode.Split('\n');
            var sLine   = () => s;
            NextLine = sLine;
            foreach (var l in forthLines)
            {
                s = l;
                Refill();
                if(Pop() == FORTH_TRUE)
                    Interpret();
            }
        } finally
        {
            NextLine = oldLine;
        }
    }
    public IEnumerable<string> Words()
    {
        List<string> userWords = new();

        var dp = dictHead;
        while(true)
        {
            if(dp == 0) break;

            var wordNameStart = dp + CELL_SIZE;
            var wordLenRaw    = ds[wordNameStart];
            var wordLen       = Utils.ResetHighBit(wordLenRaw);
            var wordSpan      = new Span<AChar>(ds, wordNameStart + 1 * CHAR_SIZE, wordLen);
            var wordName      = Encoding.UTF8.GetString(wordSpan);
            userWords.Add(wordName);

            dp = (Index)ReadCell(ds, dp);
        }
        return WordToSimpleOp.Keys.Concat(ImmediatePrimitives.Keys).Concat(userWords).OrderBy(s => s);
    }

    // TODO: clean this up, it now works because the default value of Op is 0 -> Error. It should use some kind of multidictionary here.
    Action? ImmediateAction(Token op) {
        var result =
            ImmediatePrimitives.FirstOrDefault(e => e.Value.Item1 == op);
        if(result.Value.Item1 != default)
            return result.Value.Item2;
        else
            return null;
    }

    void LowerCase()
    {
        var s  = (Index)Peek();
        var bs = new Span<byte>(ds, s + 1, ds[s]);
        foreach(ref byte b in bs)
            b = (byte) char.ToLowerInvariant((char)b); 
    }
    static void ColorLine(ConsoleColor color, string s) {
        var backupcolor = Console.ForegroundColor;
        Console.ForegroundColor = color;
        Console.WriteLine(s);
        Console.ForegroundColor = backupcolor;
    }
    internal bool IsEmptyWord() => ds[Peek()] == 0;

    public string StackString()
    {
        StringBuilder sb = new("\\ ");
        for (int i = 0; i < sp; i += CELL_SIZE)
        {
            sb.Append(ReadCell(ps, i)); sb.Append(' ');
        }
        if(sp == 0) sb.Append("empty");
        return sb.ToString();
    }
    internal void Bl()    => Push(' ');
    void State() => Push(state);

    /* These are internal to be able to test them. What a bother. */
    internal void Push(Cell n)  => ps = Utils.Add(ps, ref sp, n);
    internal Cell Pop()         => Utils.ReadBeforeIndex(ps, ref sp);
    internal Cell Peek()        => Utils.ReadCell(ps, sp - CELL_SIZE);

    internal void RPush(Cell n) => rs = Utils.Add(rs, ref rp, n);
    internal Cell RPop()        => Utils.ReadBeforeIndex(rs, ref rp);
    void PStoRS()                  => RPush(Pop());
    void FromR()                => Push(RPop());

    internal void DPush(Cell n) => ds = Utils.Add(ds, ref herep, n);

    internal void Comma()       => ds = Utils.Add(ds, ref herep, Pop());
    internal void Store()       => Utils.WriteCell(ds, (Index)Pop(), Pop());
    internal void At()          => Push(Utils.ReadCell(ds, (Index)Pop()));
    internal void Here()        => Push(herep);
    internal void Depth()       => Push(sp / CELL_SIZE);
    internal void Over()        => Push(ReadCell(ps, sp - CELL_SIZE * 2));
    internal void Swap()        { var a = Pop(); var b = Pop(); Push(a); Push(b); }

    void Char()
    {
        Bl();
        Word();
        var idx = (Index)Pop();
        if(ds[idx] == 0) Throw("'char' need more input.");
        Push(ds[idx + 1]);
    }
    internal void Refill()
    {
        if(NextLine == null) Throw("Trying to Refill, without having passed a NextLine func");

        var inputBuffer = NextLine();

        if (inputBuffer == null)
        {
            Push(FORTH_FALSE);
        }
        else
        {
            inputBuffer = inputBuffer.Trim();
            input_len_chars = Encoding.UTF8.GetByteCount(inputBuffer);
            if (input_len_chars > inputBufferSize)
                throw new Exception(
                $"Cannot parse a line longer than {inputBufferSize}. {inputBuffer} is {input_len_chars} chars long.");
            var inputCharSpan = ToChars(source, input_len_chars);
            var bytes = new Span<byte>(Encoding.UTF8.GetBytes(inputBuffer));
            bytes.CopyTo(inputCharSpan);
            WriteCell(ds, inp, 0);
            Push(FORTH_TRUE);
        }
    }
    internal void Source()
    {
        Push(source);
        Push(input_len_chars);
    }

    internal void Compare()
    {
        var u2 = (Index)Pop();
        var a2 = (Index)Pop();
        var u1 = (Index)Pop();
        var a1 = (Index)Pop();

        var s1 = new Span<AChar>(ds, a1, u1);
        var s2 = new Span<AChar>(ds, a2, u2);
        var r  = MemoryExtensions.SequenceCompareTo(s1, s2);
        Push(r < 0 ? -1 : r > 0 ? +1 : 0);
    }
    internal void Drop()  => sp -= CELL_SIZE;
    internal void Drop2() => sp -= CELL_SIZE * 2; 
    internal void Dup()   => Push(Peek());
    internal void Dup2()
    {
        var x2 = Pop();
        var x1 = Pop();
        Push(x1);
        Push(x2);
        Push(x1);
        Push(x2);
    }
    void MulDivRem()
    {
        var n3 = Pop();
        var n2 = Pop();
        var n1 = Pop();
        (var n5, var n4) = Math.DivRem(n1 * n2, n3);
        Push(n4);
        Push(n5);
    }

    void DCall()
    {
        if(lastMethod is null) Throw($"No method UNKNOWN on type {lastType.Name}");
        var args = lastMethod.GetParameters().Reverse();
        var objs =
            args.Select<ParameterInfo, object>(p => p.ParameterType == typeof(string) ? ToDotNetString() : (Cell) Pop());
        var res = lastMethod.Invoke(null, objs.Reverse().ToArray());
        if(res is null)
            Throw("null returned from this invocation.");
        else
            if(res.GetType() == typeof(string))
                FromDotNetString((string)res);
            else
                Push(Convert.ToInt64(res));
    }
    /* It is implemented like this to avoid endianess problems

Introducing Forth.Net

Wed, 23 Feb 2022 00:00:00 GMT

An implementation of the Forth programming language for the .Net Framework. You can use it as a C# library, interactively with a CLI or compile the code to a compact binary format.

It is on GitHub.

Installation

dotnet tool install -g forth.net.program

Usage

# Start the CLI.
nforth

# Interpret some files and exit.
nforth Test1.fth Test2.fth -e bye

# Compile files to binary
nforth Test1.fth Test2.fth -e 's\" myfile.io" save'

# Execute a binary file
nforth -e 's\" myfile.io" load aword'

Notes

Save the user portion of the dictionary with S" myfile.io" save. You load it with S" myfile.io" load.
Save the whole dictionary, system included, with S" myfile.io" savesys. You load it with S" myfile.io" loadsys.
To use the library, copy the file Vm.cs or reference the nuget package Forth.Net. Go here for usage notes and implementation details.
See all the supported words with words.
Perform a system test with testsys. Your results should look like the text at the bottom of this doc.
Type debug if curious.
Type nforth --help for less used options.
Call the .NET APIs with these words. (FYI just static methods with strings or numbers for now).

: escape s" System.Uri, System" s" EscapeDataString" .net ;
: sqrt   s" System.Math"        s" Sqrt"             .net ;

Test output

CR CR SOURCE TYPE ( Preliminary test ) CR
SOURCE ( These lines test SOURCE, TYPE, CR and parenthetic comments ) TYPE CR
( The next line of output should be blank to test CR ) SOURCE TYPE CR CR

( Pass #1: testing 0 >IN +! ) 0 >IN +! SOURCE TYPE CR
( Pass #2: testing 1 >IN +! ) 1 >IN +! xSOURCE TYPE CR
( Pass #3: testing 1+ ) 1 1+ >IN +! xxSOURCE TYPE CR
( Pass #4: testing @ ! BASE ) 0 1+ 1+ BASE ! BASE @ >IN +! xxSOURCE TYPE CR
( Pass #5: testing decimal BASE ) BASE @ >IN +! xxxxxxxxxxSOURCE TYPE CR
( Pass #6: testing : ; ) : .SRC SOURCE TYPE CR ; 6 >IN +! xxxxxx.SRC
( Pass #7: testing number input ) 19 >IN +! xxxxxxxxxxxxxxxxxxx.SRC
( Pass #8: testing VARIABLE ) VARIABLE Y 2 Y ! Y @ >IN +! xx.SRC
( Pass #9: testing WORD COUNT ) 5 MSG abcdef) Y ! Y ! >IN +! xxxxx.SRC
( Pass #10: testing WORD COUNT ) MSG ab) >IN +! xxY ! .SRC
Pass #11: testing WORD COUNT .MSG
Pass #12: testing = returns all 1's for true
Pass #13: testing = returns 0 for false
Pass #14: testing -1 interpreted correctly
Pass #15: testing 2*
Pass #16: testing 2*
Pass #17: testing AND
Pass #18: testing AND
Pass #19: testing AND
Pass #20: testing ?F~ ?~~ Pass Error
Pass #21: testing ?~
Pass #22: testing EMIT
Pass #23: testing S"

Results:

Pass messages #1 to #23 should be displayed above
and no error messages

0 tests failed out of 57 additional tests

Down by the seaside

Tue, 22 Feb 2022 00:00:00 GMT

The wind batters their face; the sea sings in their ears. The night is yellow with hate. I look at them from my position behind the house. They are like shadows.

She is my girlfriend. I don’t know him. They kiss, as I know they would: entwined like a grapevine. A dog barks in the distance. I jump, scared. The moon laughs at us all.

Jenny and I got together a long time ago. We were very young: our hearts pure, our desires less so. We loved each other ferociously: entwined like a grapevine.

She told me she loved me — many times. We talked about starting a family together: kids and all. We are engaged now. It is a big deal. And now this.

The rifle shines in the moonlight. It sings my revenge - the joy of the kill - the thrill of the fall.

I point the rifle.

A cloud covers the moon. I lose my target - I lose my resolve. Rain starts to fall. It washes my anger away. I breathe salt in the wind - lower my weapon - head home.

My train leaves tomorrow - a new city waits for me. Jenny is right. We are no more.

The moon shines over the rifle by the side of the house - no clouds in the sky.

I have create a font - Italiko

Wed, 23 Jun 2021 00:00:00 GMT

Here it is. Buy buy buy: https://www.myfonts.com/fonts/luca-bolognese/italiko/

Introduction to Functional Programming in F#

Sun, 23 May 2021 00:00:00 GMT

Here is a new presentation I did for Future Tech 2021. It is a gentle introduction to the core of functional programming using F# (at least the way I see it).

It is likely too simplicistic for some, but hopefully useful to many.

https://www.youtube.com/watch?v=4BfOpsx2rwI

New C# Source Generator Samples

Tue, 25 Aug 2020 00:00:00 GMT

This is just a note to keep all my material in one place.

I wrote this blog post on the MSDN .NET Blog about my CSV and Mustache C# Source Generators samples.

Text for proofing fonts

Wed, 10 Jun 2020 00:00:00 GMT

This comes from this article. It is useful to evaluate the kerning of a font. Switch the Theme at the top of the blog to see the results on the text below.

Latin Lowercase

Angel Adept Blind Bodice Clique Coast Dunce Docile Enact Eosin Furlong Focal Gnome Gondola Human Hoist Inlet Iodine Justin Jocose Knoll Koala Linden Loads Milliner Modal Number Nodule Onset Oddball Pneumo Poncho Quanta Qophs Rhone Roman Snout Sodium Tundra Tocsin Uncle Udder Vulcan Vocal Whale Woman Xmas Xenon Yunnan Young Zloty Zodiac. Angel angel adept for the nuance loads of the arena cocoa and quaalude. Blind blind bodice for the submit oboe of the club snob and abbot. Clique clique coast for the pouch loco of the franc assoc and accede. Dunce dunce docile for the loudness mastodon of the loud statehood and huddle. Enact enact eosin for the quench coed of the pique canoe and bleep. Furlong furlong focal for the genuflect profound of the motif aloof and offers. Gnome gnome gondola for the impugn logos of the unplug analog and smuggle. Human human hoist for the buddhist alcohol of the riyadh caliph and bathhouse. Inlet inlet iodine for the quince champion of the ennui scampi and shiite. Justin justin jocose for the djibouti sojourn of the oranj raj and hajjis. Knoll knoll koala for the banknote lookout of the dybbuk outlook and trekked. Linden linden loads for the ulna monolog of the consul menthol and shallot. Milliner milliner modal for the alumna solomon of the album custom and summon. Number number nodule for the unmade economic of the shotgun bison and tunnel. Onset onset oddball for the abandon podium of the antiquo tempo and moonlit. Pneumo pneumo poncho for the dauphin opossum of the holdup bishop and supplies. Quanta quanta qophs for the inquest sheqel of the cinq coq and suqqu. Rhone rhone roman for the burnt porous of the lemur clamor and carrot. Snout snout sodium for the ensnare bosom of the genus pathos and missing. Tundra tundra tocsin for the nutmeg isotope of the peasant ingot and ottoman. Uncle uncle udder for the dunes cloud of the hindu thou and continuum. Vulcan vulcan vocal for the alluvial ovoid of the yugoslav chekhov and revved. Whale whale woman for the meanwhile blowout of the forepaw meadow and glowworm. Xmas xmas xenon for the bauxite doxology of the tableaux equinox and exxon. Yunnan yunnan young for the dynamo coyote of the obloquy employ and sayyid. Zloty zloty zodiac for the gizmo ozone of the franz laissez and buzzing.

Latin Uppercase

ABIDE ACORN OF THE HABIT DACRON FOR THE BUDDHA GOUDA QUAALUDE. BENCH BOGUS OF THE SCRIBE ROBOT FOR THE APLOMB JACOB RIBBON. CENSUS CORAL OF THE SPICED JOCOSE FOR THE BASIC HAVOC SOCCER. DEMURE DOCILE OF THE TIDBIT LODGER FOR THE CUSPID PERIOD BIDDER. EBBING ECHOING OF THE BUSHED DECAL FOR THE APACHE ANODE NEEDS. FEEDER FOCUS OF THE LIFER BEDFORD FOR THE SERIF PROOF BUFFER. GENDER GOSPEL OF THE PIGEON DOGCART FOR THE SPRIG QUAHOG DIGGER. HERALD HONORS OF THE DIHEDRAL MADHOUSE FOR THE PENH RIYADH BATHHOUSE. IBSEN ICEMAN OF THE APHID NORDIC FOR THE SUSHI SAUDI SHIITE. JENNIES JOGGER OF THE TIJERA ADJOURN FOR THE ORANJ KOWBOJ HAJJIS. KEEPER KOSHER OF THE SHRIKE BOOKCASE FOR THE SHEIK LOGBOOK CHUKKAS. LENDER LOCKER OF THE CHILD GIGOLO FOR THE UNCOIL GAMBOL ENROLLED. MENACE MCCOY OF THE NIMBLE TOMCAT FOR THE DENIM RANDOM SUMMON. NEBULA NOSHED OF THE INBRED BRONCO FOR THE COUSIN CARBON KENNEL. OBSESS OCEAN OF THE PHOBIC DOCKSIDE FOR THE GAUCHO LIBIDO HOODED. PENNIES PODIUM OF THE SNIPER OPCODE FOR THE SCRIP BISHOP HOPPER. QUANTA QOPHS OF THE INQUEST OQOS FOR THE CINQ COQ SUQQU. REDUCE ROGUE OF THE GIRDLE ORCHID FOR THE MEMOIR SENSOR SORREL. SENIOR SCONCE OF THE DISBAR GODSON FOR THE HUBRIS AMENDS LESSEN. TENDON TORQUE OF THE UNITED SCOTCH FOR THE NOUGHT FORGOT BITTERS. UNDER UGLINESS OF THE RHUBARB SEDUCE FOR THE MANCHU HINDU CONTINUUM. VERSED VOUCH OF THE DIVER OVOID FOR THE TELAVIV KARPOV FLIVVER. WENCH WORKER OF THE UNWED SNOWCAP FOR THE ANDREW ESCROW GLOWWORM. XENON XOCHITL OF THE MIXED BOXCAR FOR THE SUFFIX ICEBOX EXXON. YEOMAN YONDER OF THE HYBRID ARROYO FOR THE DINGHY BRANDY SAYYID. ZEBRA ZOMBIE OF THE PRIZED OZONE FOR THE FRANZ ARROZ BUZZING.

Latin Small Caps

Abide Acorn Anaheim. Bench Bogus Babcock. Census Coral Cocoon. Demure Docile Didion. Ebbing Echoing Energetic. Feeder Focus Fiftieth. Gender Gospel Gogol. Herald Honors Hohokus. Ibsen Iceman Isinglass. Jennies Jogger Jejune. Keeper Kosher Kokopelli. Lender Locker Liliput. Menace Mccoy Mimosa. Nebula Noshed Nonesuch. Obsess Ocean Onondaga. Pennies Podium Popcorn. Quanta Qophs Queque. Reduce Rogue Reread. Senior Sconce Sesame. Tendon Torque Totality. Under Ugliness Usually. Versed Vouch Vivacious. Wench Worker Wowed. Xenon Xochitl Xerxes. Yeoman Yonder Yoyo. Zebra Zombie Zizek.

Abide abide acorn of the habit dacron for the buddha gouda quaalude. Bench bench bogus of the scribe robot for the aplomb jacob ribbon. Census census coral of the spiced jocose for the basic havoc soccer. Demure demure docile of the tidbit lodger for the cuspid period bidder. Ebbing ebbing echoing of the bushed decal for the apache anode needs. Feeder feeder focus of the lifer bedford for the serif proof buffer. Gender gender gospel of the pigeon dogcart for the sprig quahog digger. Herald herald honors of the dihedral madhouse for the penh riyadh bathhouse. Ibsen ibsen iceman of the aphid nordic for the sushi saudi shiite. Jennies jennies jogger of the tijera adjourn for the oranj kowboj hajjis. Keeper keeper kosher of the shrike bookcase for the sheik logbook chukkas. Lender lender locker of the child gigolo for the uncoil gambol enrolled. Menace menace mccoy of the nimble tomcat for the denim random summon. Nebula nebula noshed of the inbred bronco for the cousin carbon kennel. Obsess obsess ocean of the phobic dockside for the gaucho libido hooded. Pennies pennies podium of the sniper opcode for the scrip bishop hopper. Quanta quanta qophs of the inquest oqos for the cinq coq suqqu. Reduce reduce rogue of the girdle orchid for the memoir sensor sorrel. Senior senior sconce of the disbar godson for the hubris amends lessen. Tendon tendon torque of the united scotch for the nought forgot bitters. Under under ugliness of the rhubarb seduce for the manchu hindu continuum. Versed versed vouch of the diver ovoid for the telaviv karpov flivver. Wench wench worker of the unwed snowcap for the andrew escrow glowworm. Xenon xenon xochitl of the mixed boxcar for the suffix icebox exxon. Yeoman yeoman yonder of the hybrid arroyo for the dinghy brandy sayyid. Zebra zebra zombie of the prized ozone for the franz arroz buzzing.

Recursively extracting nested links from a webpage using Racket

Mon, 11 May 2020 00:00:00 GMT

This is my first attempt at writing Racket or LISP code. It might be ugly … For now, let’s build a web crawler, next, I shall write myself a language. That’s where Racket really shine.

The code is here. Thanks to Mike for reviewing it.

Why

I want to translate a website, including recursively reached pages, to a pdf to read on my e-reader. This program does the first step: going from a URL to a list of URLs by recursively navigating the links on the page.

The PS1Script directory contains scripts to perfrom the other steps: 1. Translate all the links to PDF pages (using Chrome headless). 2. Combine all the PDFs to a single document (using cpdf).

Prelude

First you declare which language you are using. Racket is a language to build languages and the language itself is just one of many (try rereading this phrase).

#lang racket

We then expose the main functions of the library. Aka recursively getting all links from a webpage to a certain nesting level both as a Racket list and as a newline delimitated string.

; Provides functions to extract links from web pages recursively

(provide
 ; uriString nestingLevels -> list of uriStrings
 ; Doesn't follow links to non-html resources or pointing to a different domain then uriString.
 uri->nestedLinks
 ; uriSTring nestingLevels -> string of newline separated Uri Strings
 uri->nestedLinksNl)

Implementation

This is similar, but not identical, to C# using statement as you declare which packages are needed.

(require (planet neil/html-parsing:2:0)
         net/url
         xml
         html
         sxml/sxpath
         threading)

The program crawl just links to html files in the same domain as the first URI given to it. It is possible to extend this more by having the program take a regular expression (or * expression) to identify which file sto leave out of the crawling.

(define invalid-suffixes
  '("./" ".xml" ".jpg" ".jpeg" ".png" ".gif" ".tiff" ".psd"
    ".eps" ".ai" ".indd" ".raw" ".svg"))

(define invalid-prefixes '("#" "mailto:" "javascript:"))

(define (different-domain? baseUrl l)
  (define url (string->url l))
  (and (url-host url) (not (equal? (url-host baseUrl) (url-host url)))))

(define (good-link? baseUrl l) (not (or (different-domain? baseUrl l)
                                (ormap (curry string-suffix? l) invalid-suffixes)
                                (ormap (curry string-prefix? l) invalid-prefixes))))

Next we have to parse the HTML. We use XPath for that. Racket is particularly good for XML parsing as it maps naturally to expressions which are the bread and butter of the language. I don’t enjoy the nesting of expressions that LISP like languages force onto you (as in the expression below). But see later for a partial solution.

(define (xexp->links xexp) (flatten (map cdr ((sxpath "//a/@href") xexp))))

The strange ~>> operator in the code below comes from the Threading package. This set of macros lets you build pipelines of computations similarly to the F# |> operator. The initial value comes first and then the functions are applied in series each one to the result of the previous. By doing that you ‘flatten’ the nested expressions, making them more readable (at least to my eyes).

This capability of changing the core behavior of the language is something very peculiar to the LISP family, and the reaason why I am attracted to Racket in the first place.

This function extracts all the ‘good’ links from a url. BTW: I love that you can use -> to name symbols.

(define (url->links url)
  (~>> (call/input-url url get-pure-port html->xexp)
       xexp->links
       (filter (curry good-link? url))))

λ~> is the function composition operator in the ‘Threading’ library. You got to love embedding lambdas in the code.

(define uri->links
  (λ~> string->url url->links))

This is the main recursive workhorse of the program. It works something like this (numbers marked in the code):

Treat links to subparts of a web page as if they were links to the webpage
If it is not a good link, return the links already visited (visited)
Same thing if the link is alread in visited
If we reached the nesting level specified, add the link to visited and return ‘visited’
Otherwise add the link to visited and call yourself on all sublinks on the page

The function is not tail recursive, but that is not a huge deal in Racket as the stack is very large. It doesn’t blow up as easily as in most other languages.

(define (uri->nestedLinks-rec baseUrl uri visited levels)

  (define abs-url (if (string-contains? uri "#") ; <0>
      (~> uri (string-split "#") first (combine-url/relative baseUrl _))
      (~>> uri (combine-url/relative baseUrl))))

  (log-info "~a, ~a, ~a:~a~n" (url->string baseUrl)
                              levels uri (url->string abs-url))

  (cond [(not (good-link? baseUrl uri)) visited] ; <1>
        [(member abs-url visited)  visited]      ; <2>
        [(zero? levels)  (cons abs-url visited)] ; <3>
        [else  (for/fold ([acc (cons abs-url visited)]) ;<4>
                         ([l (in-list (url->links abs-url))])
                 (uri->nestedLinks-rec abs-url l acc (sub1 levels)))]))

Finally we can trivially define the two main functions in the module.

(define (uri->nestedLinks uri levels)
  (reverse (uri->nestedLinks-rec (string->url uri) "" '() levels)))

(define (uri->nestedLinksNl uri levels)
  (define links (uri->nestedLinks uri levels))
  (string-join (map url->string links) "\n" #:after-last "\n"))

Test

To my great pleasure, Racket allows (encourages?) you to have tests in the same file as the code. They just go into sub modules, that can be constructed piecewise with the module+ instruction.

You could add the tests beside each function, but I decided to have a separate section in the file instead. To run them you call raco test FILENAME.

(define (uri->path test-uri)
        (build-path "./data" (~> test-uri first uri->file string->path)))
(define uri->file (λ~> string->url url-host))
(define test-uris '(
                    ("https://www.lucabol.com" 3)
                    ("https://beautifulracket.com/" 3)
                    ("https://en.wikipedia.org/wiki/Typeface" 1)
                    ("https://brieferhistoryoftime.com" 3)
                    ("https://mobydick.wales/" 3)
                    ("https://resilientwebdesign.com" 3)
                    ("https://www.c82.net/euclid/" 3)
                    ))

(module+ test
  (require rackunit)

I got a bit sloppy not naming my lambdas here … But, doesn’t the lambda symbol look cool?

  (for-each (λ (test-uri)
              (with-input-from-file
                  (uri->path test-uri)
                (λ () (begin
                        (define saved-result (port->string))
                        (define calc-result
                          (uri->nestedLinksNl (first test-uri) (second test-uri)))
                        (check-equal? calc-result saved-result test-uri)))
                #:mode 'text
                ))
            test-uris))

This is used to regenerate the test data. You can then inspect it manually before running tests.

(define (refresh-test-data)
    (for-each (λ (test-uri)
                (with-output-to-file
                    (uri->path test-uri)                  
                  (λ () (display (uri->nestedLinksNl (first test-uri) (second test-uri))))
                  #:exists 'replace))
              test-uris))

Main

Main goes into its own submodule as well. Racket is not as pure as Haskell, so you can naturally manage side effects like user input and such. You got to appreciate the concisivness of the command line parsing library.

The code below looks a bit odd to me. It could probably be refactored so that the parser expression returns the values instead of filling out parameters.

(module+ main

  (define levels (make-parameter "3"))
  (define uri (make-parameter #f))

  (define parser
    (command-line
     #:program "website-links"
     #:usage-help "Extracts links from a webpage recursively to a specified level."
     #:once-each
     [("-l" "--levels") LEVELS "How many nested levels to process (default 3)." (levels LEVELS)]
     #:args (URI) (uri URI)))

  (display (uri->nestedLinksNl (uri) (string->number (levels)))))

Conclusion

I liked Racket very much. It takes a little while to get use to the expression syntax, which is very different from the C-like one most of us are used to. It also takes a while to get used to the style of the documentation, which is written very precisely for the careful reader. We are more used to the ‘here is an example, copy it’ kind of documentation. For the distracted programmer …

Confusing characters in text

Sun, 19 Jan 2020 00:00:00 GMT

In the process of chosing fonts for my blog, I looked for lists of words containing characters that can be easily confused.

I took this one from here.

Henry III - Henry lII - Henry 111 - Henry tll - Henry t7I

Henry l - Henry 1 - Henry I - Henry 7

i j, B 8, D 0, O 0, o 0, k R, a o, f F , r v Y, B 8, g q, a g, J I, J L, l !, u v, U V, 5 S, B 8, a o, rn m, vv w, Z 2, C G, Q O, ISO 150, go 90, LJ IJ LI Ll L1 U, LU LLI UU W, LILI UU, CI O 0, <j (J (j (T

clear dear, burn bum, skivvy skiwy, CS5 CSS, 105 IOS, z2 22, 5AM SAM, travel bavel, LJ U, MENSUR SULJOVIC, 6 9, ce œ, amazon.corn amazon.com mac.com rnac.com, Ox0 0OR

minnimalistI1llI

Henry III - Henry lII - Henry 111 - Henry tll - Henry t7I

Henry l - Henry 1 - Henry I - Henry 7

i j, B 8, D 0, O 0, o 0, k R, a o, _f_ _F_ , r v Y, B 8, g q, a g, J I, J L, l !, u v, U V, 5 S, B 8, a o, rn m, vv w, Z 2, C G, Q O, ISO 150, go 90, LJ IJ LI Ll L1 U, LU LLI UU W, LILI UU, CI O 0, <j (J (j (T

clear dear, burn bum, skivvy skiwy, CS5 CSS, 105 IOS, z2 22, 5AM SAM, travel bavel, LJ U, MENSUR SULJOVIC, 6 9, ce œ, amazon.corn amazon.com mac.com rnac.com, Ox0 0OR 

minnimalistI1llI

Introducing PS1 Libraries for Azure Lab Services and DevTest Labs

Mon, 06 Jan 2020 00:00:00 GMT

I am not sure why I have not blogged about this. We have released some very nice PS1 libraries to automate the usage of Azure Lab Services and Azure DevTest Labs.

The nicest thing about them is how they make use of the PS1 Pipeline to give a nice user experience.

Get-AzLabAccount | Get-AzLab | Get-AzLabVm -Status Running | Stop-AzLabVm

I intend to shoot some brief videos to describe how to use them. In the meantime, you can read tutorials here and here

Status of C# 8.0 functional features with a comparison to Haskell

Fri, 06 Dec 2019 00:00:00 GMT

Abstract

Writing C# functional code has become easier with each new release of the language (i.e. nullable ref types, tuples, switch expr, …). This document presents a review of the current status of basic functional features for C# 8.0. It focuses mostly on syntax and aims to achieve its goal using code examples. It doesn’t touches on more advanced topics as Monad, Functors, etc …

Haskell has been chosen as the ‘comparison’ language (using few examples from here and elsewhere). This is not intended as a statement of value of one language vs the other. The languages are profoundly different in underlying philosophies. They both do much more than what is presented here (i.e. C# supports OO and imperative style, while Haskell goes much deeper in type system power etc…).

I also present samples of usage of language-ext to show what can be achieved in C# using a functional library.

Using a library of functions

In C# you can use static functions as if they were ‘floating’ by importing the static class they are defined to with the syntax using static CLASSNAME. Below an example of both importing Console.WriteLine and using it.

using System;
using System.IO;

using LanguageExt;
using LanguageExt.TypeClasses;
using LanguageExt.ClassInstances;
using System.Collections.Generic;

using static System.Console;
using static System.Linq.Enumerable;
using static System.Math;
using static LanguageExt.Prelude;

public static class Core {

    static void UseFunc() => WriteLine("System.WriteLine as a floating function");

Writing simple functions

In Haskell, a simple function might be written as:

   square :: Num a => a -> a
   square x = x * x

Which in C# looks like the below.

    static int Square(int x) => x * x;

As an aside, note that we lose the generality of the function (i.e. we need a different one for doubles).This is due to the lack of ad-hoc polymorphism in C#. By using language-ext, you can fake it so that it looks like this:

    static A Square<NumA, A>(A x) where NumA : struct, Num<A> => default(NumA).Product(x, x);

    static void SquarePoly() {
        WriteLine(Square<TInt, int>(2));
        WriteLine(Square<TDouble, double>(2.5));
    }

In Haskell, it is conventional to write the type of a function before the function itself. In C# you can use Func and Action to achieve a similar goal (aka seeing the types separately from the function implementation), despite with more verbose syntax, as below:

    static Func<int, int> SquareF = x => x * x;

Pattern matching

Here is an example in Haskell:

    lucky :: (Integral a) => a -> String  
    lucky 7 = "LUCKY NUMBER SEVEN!"  
    lucky x = show x

In C#, you can write it in a few ways. Either as a function static property …

    static Func<int, string> Lucky = x => x switch {
            7 => "LUCKY NUMBER SEVEN!",
            _ => x.ToString()
        };

Or with a normal function:

    static string Lucky1(int x) => x switch {
            7 => "LUCKY NUMBER SEVEN!",
            _ => x.ToString()
    };

Or even using the ternary operator (a perennial favorite of mine).

    static string Lucky2(int x) => x == 7 ? "LUCKY NUMBER SEVEN!"
                                          : x.ToString();

Pattern matching on tuples works with similar syntax:

    static bool And(bool x, bool y) =>
        (x, y) switch
        {
            (true, true) => true,
            _            => false
        };

In Haskell you can match on lists as follows:

    sum :: Num a => [a] -> a
    sum [] = 0
    sum (x:xs) = x + sum xs

Writing this in standard C# looks like this:

    static int Sum(IEnumerable<int> l) => l.Count() == 0
                                            ? 0
                                            : l.First() + Sum(l.Skip(1));

    static int Sum1(IEnumerable<int> l) => l.Count() switch {
        0 => 0,
        _ => l.First() + Sum(l.Skip(1))
    };

Language-ext gives you a simpler syntax, with more flexibility on what you can match against:

    static int Sum2(Seq<int> l) =>
        match(l,
               () => 0,
               (x, xs) => x + Sum1(xs));

Obviously you always have to be careful with recursion in C# (here). Better use the various methods on Enumerable.

Guards (and case expressions)

Let’s explore guards. Case expressions have an identical translation in C#.

In Haskell guards are used as below:

    bmiTell :: (RealFloat a) => a -> a -> String  
    bmiTell weight height  
            | weight / height ^ 2 <= 18.5 = "Under"  
            | weight / height ^ 2 <= 25.0 = "Normal"  
            | weight / height ^ 2 <= 30.0 = "Over"  
            | otherwise                   = "Way over"

Which can be coded in C# as:

    static string BmiTell(double weight, double height) =>
        (weight, height) switch
        {
            _ when Pow(weight / height, 2) <= 18.5 => "Under",
            _ when Pow(weight / height, 2) <= 25.0 => "Normal",
            _ when Pow(weight / height, 2) <= 30.0 => "Over",
            _                                      => "Way over"
        };

Obviously this is quite bad. You would like something more like:

    bmiTell :: (RealFloat a) => a -> a -> String  
    bmiTell weight height  
            | bmi <= skinny = "Under"  
            | bmi <= normal = "Normal"  
            | bmi <= fat    = "Over"  
            | otherwise     = "Way Over"  
            where bmi = weight / height ^ 2  
                  skinny = 18.5  
                  normal = 25.0  
                  fat = 30.0

But it is not trivial in C# to declare variables in expression bodied members. You can either move it to a normal method or abuse the LINQ query syntax. Both shown below. Notice that this is more similar to let expressions in Haskell, as they come before the expression, not after.

    static string BmiTell1(double weight, double height) {
        double bmi = Pow(weight / height, 2),
               skinny = 18.5,
               normal = 25.0,
               fat = 30;

        return bmi switch
        {
            _ when bmi <= skinny => "Under",
            _ when bmi <= normal => "Normal",
            _ when bmi <= fat    => "Over",
            _                    => "Way over"
        };
    }

    static string BmiTell2(double weight, double height) =>
        (   from _ in "x"
            let bmi     = Pow(weight / height, 2)
            let skinny  = 18.5
            let normal  = 25.0
            let fat     = 30
            select   bmi <= skinny ? "Under"
                   : bmi <= normal ? "Normal"
                   : bmi <= fat    ? "Over"
                   :                 "Way over").First();

Product types (aka Records)

In Haskell you define a product type as below:

    data Person = Person { firstName :: String  
                             , lastName :: String  
                             , age :: Int
                             } deriving (Show)

In C#, it is currently complicated to define an immutable product type with structural equality, structural ordering and efficient hashing.

In essence, you have to implement a bunch of interfaces and operators, somehow similar to below (and I am not implementing ordering, and it is probably not too efficient either).

    public readonly struct PersonData: IEquatable<PersonData> {
        public readonly string FirstName;
        public readonly string LastName;
        public readonly int Age;

        public PersonData(string first, string last, int age) => (LastName, FirstName, Age) = (last, first, age);

        public override int GetHashCode() => (FirstName, LastName, Age).GetHashCode();
        public override bool Equals(object other) => other is PersonData l && Equals(l);
        public bool Equals(PersonData oth) => LastName == oth.LastName && FirstName == oth.FirstName && Age == oth.Age;
        public static bool operator ==(PersonData lhs, PersonData rhs) => lhs.Equals(rhs);
        public static bool operator !=(PersonData lhs, PersonData rhs) => !(lhs == rhs);
    }

If you use a struct, you are still open to someone simply newing it using the default constructor, that you can’t make private.

Using a class (as below) avoids that, but loses the pass by value semantic.

    public class PersonData1 : IEquatable<PersonData1> {
        public readonly string FirstName;
        public readonly string LastName;
        public readonly int Age;

        public PersonData1(string first, string last, int age) => (LastName, FirstName, Age) = (last, first, age);

        public override int GetHashCode() => (FirstName, LastName, Age).GetHashCode();
        public override bool Equals(object oth) => oth is PersonData l && Equals(l);
        public bool Equals(PersonData1 other) => LastName == other.LastName && FirstName == other.FirstName && Age == other.Age;
        public static bool operator ==(PersonData1 lhs, PersonData1 rhs) => lhs.Equals(rhs);
        public static bool operator !=(PersonData1 lhs, PersonData1 rhs) => !(lhs == rhs);
    }

So, there is no easy fix. Using Language-ext you can do it much more simply, by inheritance. But obviously it uses IL generation that is slow the first time around. Try running the code and notice the delay when IL generating.

    public class PersonData2 : Record<PersonData2> {
        public readonly string FirstName;
        public readonly string LastName;
        public readonly int Age;

        public PersonData2(string first, string last, int age) => (LastName, FirstName, Age) = (last, first, age);

    }

Sum types (aka Discriminated Union)

In Haskell you write:

    data Shape =
                Circle Float Float Float
            | Rectangle Float Float Float Float
            | NoShape
            deriving (Show)

There is no obvious equivalent in C#, and different libraries has sprung up to propose possible solutions (but not language-ext) (i.e. here or here).

One possible ‘pure language’ implementation, not considering structural equality/ordering/hash, follows:

    abstract class Shape {

        public sealed class NoShape : Shape { }
        public sealed class Circle : Shape {
            internal Circle(double r) => Radius = r;
            public readonly double Radius;
        }
        public sealed class Rectangle : Shape {
            internal Rectangle(double height, double width) => (Height, Width) = (height, width);
            public readonly double Height;
            public readonly double Width;
        }
    }

    static Shape.Circle Circle(double x)                 => new Shape.Circle(x);
    static Shape.Rectangle Rectangle(double x, double y) => new Shape.Rectangle(x,y);
    static Shape.NoShape NoShape()                       => new Shape.NoShape();

You can then pattern match on it in various obvious ways:

    static double Area(Shape s) => s switch
    {
        Shape.NoShape _                  => 0,
        Shape.Circle {Radius: var r }    => Pow(r, 2) * PI,
        Shape.Rectangle r                => r.Height * r.Width,
        _                                => throw new Exception("No known shape")
    };

    static void CalcAreas() {
        var c = Circle(10);
        var r = Rectangle(10, 3);
        var n = NoShape();

        WriteLine(Area(c));
        WriteLine(Area(r));
        WriteLine(Area(n));
    }

Maybe (Or Option) type

In Haskell you write:

     f::Int -> Maybe Int
     f 0 = Nothing
     f x = Just x

     g::Maybe Int -> Int
     g Nothing  = 0
     g (Just x) = x

In C#, this easily translates to Nullable value and reference types. Assume you have Nullable = enabled in your project

    static int? F(int i) => i switch {
        0 => new Nullable<int>(),
        _ => i
    };

    static int G(int? i) => i ?? 0;

Main Method

Let’s wrap all the samples with a Main function and then write a full program.

    static void Main() {
        UseFunc();
        SquarePoly();
        WriteLine(Square(2) == SquareF(2));
        WriteLine(Sum(new[] { 1, 2, 3, 4 })  == Sum1(new[] { 1, 2, 3, 4 }));
        WriteLine(Sum1(new[] { 1, 2, 3, 4 }) == Sum2(new[] { 1, 2, 3, 4 }.ToSeq()));
        WriteLine(BmiTell(80, 100) == BmiTell1(80, 100));
        WriteLine(BmiTell(80, 100) == BmiTell2(80, 100));

        WriteLine(new PersonData("Bob", "Blake", 40) == new PersonData("Bob", "Blake", 40));
        WriteLine(new PersonData1("Bob", "Blake", 40) == new PersonData1("Bob", "Blake", 40));
        WriteLine("Before IL gen");
        WriteLine(new PersonData2("Bob", "Blake", 40) == new PersonData2("Bob", "Blake", 40));
        WriteLine(new PersonData2("Alphie", "Blake", 40) <= new PersonData2("Bob", "Blake", 40));
        WriteLine("Already genned");
        WriteLine(new PersonData2("Bob", "Blake", 40) == new PersonData2("Bob", "Blake", 40));
        WriteLine(new PersonData2("Alphie", "Blake", 40) <= new PersonData2("Bob", "Blake", 40));

        CalcAreas();

        Hangman.Core.MainHangman();

    }
}

Full program

Let’s finish with a semi-working version of Hangman, from an exercise in Haskell programming from first principles, just to get an overall impression of how the two languages look for bigger things.

The exercise was a fill-in-the-blanks kind of thing with the function names and types given, so I don’t think I butcher it too badly, but maybe not.

The Haskell code is:

module Main where

import Control.Monad (forever) -- [1]
import Data.Char (toLower) -- [2]
import Data.Maybe (isJust) -- [3]
import Data.List (intersperse) -- [4]
import System.Exit (exitSuccess) -- [5]
import System.Random (randomRIO) -- [6]

type WordList = [String]

allWords :: IO WordList
allWords = do
      dict <- readFile "data/dict.txt"
      return (lines dict)

minWordLength :: Int
minWordLength = 5

maxWordLength :: Int
maxWordLength = 9

gameWords :: IO WordList
gameWords = do
      aw <- allWords
      return (filter gameLength aw)
      where gameLength w =
              let l = length (w :: String)
              in l > minWordLength && l < maxWordLength

randomWord :: WordList -> IO String
randomWord wl = do
      randomIndex <- randomRIO ( 0, length wl - 1)
      return $ wl !! randomIndex

randomWord' :: IO String
randomWord' = gameWords >>= randomWord

data Puzzle = Puzzle String [Maybe Char] [Char]

instance Show Puzzle where
      show (Puzzle _ discovered guessed) =
        (intersperse ' ' $ fmap renderPuzzleChar discovered)
        ++ " Guessed so far: " ++ guessed

freshPuzzle :: String -> Puzzle
freshPuzzle s = Puzzle s (map (const Nothing) s) []

charInWord :: Puzzle -> Char -> Bool
charInWord (Puzzle s _ _) c = c `elem` s

alreadyGuessed :: Puzzle -> Char -> Bool
alreadyGuessed (Puzzle _ _ s) c = c `elem` s

renderPuzzleChar :: Maybe Char -> Char
renderPuzzleChar Nothing  = '_'
renderPuzzleChar (Just c) = c

fillInCharacter :: Puzzle -> Char -> Puzzle
fillInCharacter (Puzzle word filledInSoFar s) c =
      Puzzle word newFilledInSoFar (c : s)
      where zipper guessed wordChar guessChar =
              if wordChar == guessed
                then Just wordChar
                else guessChar
            newFilledInSoFar =
              zipWith (zipper c) word filledInSoFar

handleGuess :: Puzzle -> Char -> IO Puzzle
handleGuess puzzle guess = do
      putStrLn $ "Your guess was: " ++ [guess]
      case (charInWord puzzle guess, alreadyGuessed puzzle guess) of
        (_, True) -> do
          putStrLn "You already guessed that character, pick something else!"
          return puzzle
        (True, _) -> do
          putStrLn "This character was in the word,filling in the word accordingly"
          return (fillInCharacter puzzle guess)
        (False, _) -> do
          putStrLn "This character wasn't in the word, try again."
          return (fillInCharacter puzzle guess)

gameOver :: Puzzle -> IO ()
gameOver (Puzzle wordToGuess _ guessed) =
      if (length guessed) > 7 then do
          putStrLn "You lose!"
          putStrLn $ "The word was: " ++ wordToGuess
          exitSuccess
      else return ()

gameWin :: Puzzle -> IO ()
gameWin (Puzzle _ filledInSoFar _) =
      if all isJust filledInSoFar then do
        putStrLn "You win!"
        exitSuccess
      else return ()

runGame :: Puzzle -> IO ()
runGame puzzle = forever $ do
      gameOver puzzle
      gameWin puzzle
      putStrLn $ "Current puzzle is: " ++ show puzzle
      putStr "Guess a letter: "
      guess <- getLine
      case guess of
        [c] -> handleGuess puzzle c >>= runGame
        _ -> putStrLn "Your guess must be a single character"


main :: IO ()
main = do
      word <- randomWord'
      let puzzle = freshPuzzle (fmap toLower word)
      runGame puzzle

Which loosely translate to the code below (pure C#, no language-ext) . A few comments:

I tried to keep the translation as 1:1 as possible.
I used expression bodied members for everything except IO returning function. That’s a pleasing convention to me.
Things translate rather straightforwardly except for:
1. Cheated using a simple struct instead of a Record, but often it is ok to do so.
2. Needed to use LINQ query syntax to translate more complex expressions, but cannot have lambadas in it.
3. Needed to do manual currying (language-ext would have beautified that).
The line count for this example is roughly similar. I think that’s random. Also the C# code ‘extends to the right’ more.
Notably absent from the code are sum types, which would have been verbose to implement in C#.

namespace Hangman {
    using WordList = IEnumerable<String>;
    using static System.Linq.Enumerable;

    static class Core {

        const int MinWordLength = 5;
        const int MaxWordLength = 9;

        static WordList AllWords => File.ReadAllLines("data/dict.txt");

        static WordList GameWords =>
            AllWords.Where(w => w.Length > MinWordLength && w.Length < MaxWordLength);

        static Random r = new Random();

        static string RandomWord(WordList wl) => GameWords.ElementAt(r.Next(0, wl.Length()));

        static string RandomWord1 => RandomWord(GameWords);

        static char RenderPuzzleChar(char? c) => c ?? '_';

        struct Puzzle { // Not implemented Eq and Ord because not needed in this program
            public string Word;
            public IEnumerable<char?> Discovered;
            public string Guessed;

            public override string ToString() =>
                $"{string.Join(" ", Discovered.Select(RenderPuzzleChar))}"
                    + " Guessed so far: " + Guessed;
        }

        static Puzzle FreshPuzzle(string s) => new Puzzle {
            Word       = s,
            Discovered = s.Select(_ => new Nullable<char>()),
            Guessed    = ""
        };

        static bool CharInWord(Puzzle p, char c)     => p.Word.Contains(c);
        static bool AlreadyGuessed(Puzzle p, char c) => p.Guessed.Contains(c);

        // Can't assign lambda expression to range variable with let, hence separate function
        static char? Zipper(char guessed, char wordChar, char? guessChar) =>
            wordChar == guessed ? wordChar : guessChar;

        // Manual curry. Could use language-ext to make it more beautiful.
        static Func<char, char?, char?> Zipper1(char c) => (c1, c2) => Zipper(c, c1, c2);

        static Puzzle FillInCharacter(Puzzle p, char c) =>
            (from _ in "x"
             let newFilledInSoFar = System.Linq.Enumerable.Zip<char, char?, char?>(p.Word, p.Discovered, Zipper1(c))
             select new Puzzle {
                Word = p.Word,
                Discovered = newFilledInSoFar,
                Guessed = c + p.Guessed
            }).First();

        static Puzzle HandleGuess(Puzzle puzzle, char guess) {// Braces means IO ...
            WriteLine($"Your guess was {guess}");
            switch (CharInWord(puzzle, guess), AlreadyGuessed(puzzle, guess)) {
                case (_, true):
                    WriteLine("You already guessed that character, pick something else!");
                    return puzzle;
                case (true, _):
                    WriteLine("This character was in the word,filling in the word accordingly");
                    return FillInCharacter(puzzle, guess);
                case (false, _):
                    WriteLine("This character wasn't in the word, try again.");
                    return FillInCharacter(puzzle, guess);
            }
        }

        static void GameOver(Puzzle p) {
            if(p.Guessed.Length > 7) {
                WriteLine("You lose!");
                WriteLine($"The word was {p.Word}");
                Environment.Exit(0);
            }
        }

        static void GameWin(Puzzle p) {
            if(p.Discovered.All(c => c.HasValue)) {
                WriteLine("You win!");
                Environment.Exit(0);
            }
        }

        static void RunGame(Puzzle puzzle) {
            while(true) {
                GameOver(puzzle);
                GameWin(puzzle);
                WriteLine($"Current puzzle is: {puzzle}");
                WriteLine("Guess a letter: ");
                var guess = ReadLine();
                if(guess.Length == 1) RunGame(HandleGuess(puzzle, guess[0]));
                else WriteLine("Your guess must be a single char");
            }
        }

        public static void MainHangman() {
            var puzzle = FreshPuzzle(RandomWord1);
            RunGame(puzzle);
        }

    }
}

A single .NET Core project to rule them all

Mon, 04 Feb 2019 00:00:00 GMT

Abstract

The code is here.

Thanks to Mike for reviewing this.

I have always been mildly irritated by how many .net projects I need to create in my standard workflow.

Usually I start with an idea for a library; I then want to test it with a simple executable; write some XUnit tests for it and finally benchmark some key scenarios. So I end up with at least four projects to manage.

Sure, I can find ways to automatically generate those projects, but I have always been weary of codegen to solve complexity issues. It always ends up coming back to bite you. For those of you as old as I am, think MFC …

So what is my ideal world then? Well, let’s try this:

One single project for the library and related artifacts (i.e. test, benchmarks, etc…).
Distinguish the library code from the test code from the benchmark code by some convention (i.e. name scheme).
Generate each artifact (i.e. library, tests, benchmarks, executable) by passing different options to dotnet build and dotnet run.
Create a new project by using the standard dotnet new syntax.
Have intellisense working normally in each file for my chosen editor (VSCode).
Work with dotnet watch so that one can automatically run tests when anything changes.

Disclaimer

What follows, despite working fine, is not the standard way .net tools are used. It is not in the ‘golden path’. That is problematic for production usage as:

It might not work in your particular configuration.
It might not work with other tools that rely on the presence of multiple projects (i.e. code coverage? …).
It might work now in all scenarios, but get broken in the future as you update to a new framework, sdk, editor.
It might expose bugs in the tools, now or later, which aren’t going to be fixed, as you are not using the tools as intended.
It might upset your coworkers that are used to a more standard setup.

I need to write a blog post about the concept of the ‘golden path’ and the perils, mostly hidden, of getting away from it. The summary, it is a bad idea.

Having said all of that, for the daring souls, here is one way to achieve most of the above. It also works out as a tutorial on how the different components of the .NET Core build system interacts.

How to use it

Here are the steps:

Type dotnet new -i Lucabol.SingleSourceProject.
Create a directory for your project and cd to it.
Type dotnet new lsingleproject and optionally --standardVersion <netstandardXX> --appVersion <netcoreappXX>.
Either modify the Library.cs, Main.cs, Test.cs, Bench.cs files or create your own with this convention:
- Code for the executable goes in potentially multiple files named XXXMain.cs (i.e. MyLibrary.Main.cs).
- Code for the tests goes into files named XXXTest.cs (i.e. MyLibrary.Test.cs).
- Code for the benchmarks goes into files named XXXBench.cs (i.e. MyLibrary.Bench.cs).
- Any .cs file not following the above conventions is compiled into the dll.
Type:
- dotnet build or dotnet build -c release to build debug or release version of your dll. This doesn’t include any of the main, test or bench code.
- dotnet build -c main or dotnet build -c main_release and the corresponding dotnet run -c .. build and run the exe.
- dotnet build -c test, dotnet build -c test_release and dotnet test -c test build and run the tests.
- dotnet build -c bench, dotnet run -c bench build and run the benchmark.

How it all works

The various steps above are implemented as follows:

dotnet new -i ... install a custom template that I have created and pushed on NuGet.

The custom template is composed of the following files:

Code files

There is one file for each kind of artifact that the project can generate: library, program, tests and benchmark. The files follow the name terminating conventions, as described above.

Project file

The project file is identical to any other project file generated by dotnet new except that there is one additional line appended at the end:

<Import Project="Base.targets" />

This instruct msbuild to include the Base.targets file. That file has most of the magick. I have separated it out so that you can use it unchanged in your own projects.

Base.targets

We start by removing all the file from compilation except the ones that are used to build the library.

  <ItemGroup>
    <Compile Remove="**/*Bench.cs;**/*Test.cs;**/*Main.cs" />
  </ItemGroup>

We then conditionally include the correct ones depending on which configuration is chosen. Please notice the last line, which instruct dotnet watch to watch all the .cs files. By default it just watches the ones in the debug configuration.

  <ItemGroup>
    <Compile Include="**/*Test.cs"  Condition="'$(Configuration)'=='Test'"/>
    <Compile Include="**/*Test.cs"  Condition="'$(Configuration)'=='Test_Release'"/>
    <Compile Include="**/*Bench.cs" Condition="'$(Configuration)'=='Bench'"/>
    <Compile Include="**/*Main.cs"  Condition="'$(Configuration)'=='Main'"/>
    <Compile Include="**/*Main.cs"  Condition="'$(Configuration)'=='Main_Release'"/>
    <Watch Include="**\*.cs" />
  </ItemGroup>

Then we need to define the references. Depending on what you are building you need to include references to the correct NuGet packages (i.e. if you are building test you need the xunit packages). This is done below:

  <ItemGroup  Condition="'$(Configuration)'=='Bench' OR '$(Configuration)'=='Debug'">
    <PackageReference Include="BenchmarkDotNet" Version="0.11.3" />
  </ItemGroup>

  <ItemGroup Condition="'$(Configuration)'=='Test' OR '$(Configuration)'=='Test_Release' OR '$(Configuration)'=='Debug'">
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.9.0" />
    <PackageReference Include="xunit" Version="2.4.0" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.4.0" />
  </ItemGroup>

One thing to notice is that most references are also included in the debug configuration. This is not a good thing, but it is the only way to get VSCode Intellisense to work for all the files in the solution. Apparently, IntelliSense uses whatever reference are defined for the debug build in VsCode. So debug is special, if you wish …

But that’s not enough. When you create your own MsBuild configurations, you also have to replicate the properties and constants that are set in the debug and release configurations. You would like a way to inherit them, but I don’t think it is possible.

It is particularly important to set the TargetFramework property, as it needs to be set to netcoreappXXX for the main, test and benchmark configurations. I give an example of the Test and Test_release configurations below. The rest is similar:

  <PropertyGroup Condition="'$(Configuration)'=='Test'">
    <TargetFramework>netcoreapp2.1</TargetFramework>
    <DefineConstants>$(DefineConstants);DEBUG;TRACE;TEST</DefineConstants>
    <DebugSymbols>true</DebugSymbols>
    <DebugType>portable</DebugType>
    <Optimize>false</Optimize>
  </PropertyGroup>

  <PropertyGroup Condition="'$(Configuration)'=='Test_Release'">
    <TargetFramework>netcoreapp2.1</TargetFramework>
    <DefineConstants>$(DefineConstants);RELEASE;TRACE;TEST</DefineConstants>
    <DebugSymbols>false</DebugSymbols>
    <DebugType>portable</DebugType>
    <Optimize>true</Optimize>
  </PropertyGroup>

The .template.config/template.json file

This is necessary to create a dotnet new custom template. The only thing to notice is the two parameters standardVersion and appVersion that gives the user a way to indicate which version of the .NET Standard to use for the library and which version of the application framework to use for Main, Test and Bench.

{
  "$schema": "http://json.schemastore.org/template",
  "author": "Luca Bolognese",
  "classifications": [ "Classlib", "Console", "XUnit" ],
  "identity": "Lucabol.SingleSourceProject",
  "name": "One single Project",
  "description": "One single Project for DLL, XUnit, Benchmark & Main, using configurations to decide what to compile",
  "shortName": "oneproject",
  "tags": {
    "language": "C#",
    "type": "project"
  },
  "preferNameDirectory": true,
  "sourceName": "SingleSourceProject",
  "symbols":{
    "standardVersion": {
      "type": "parameter",
      "defaultValue": "netstandard2.0",
      "replaces":"netstandard2.0"
    },
    "appVersion": {
      "type": "parameter",
      "defaultValue": "netcoreapp2.1",
      "replaces":"netcoreapp2.1"
    }
  }
}

Conclusion

Now that you know how it all works, you can make an informed decision if to use it or not. As for me …

Meta-programming in C# with JIT dead code removal and inlining

Tue, 29 Jan 2019 00:00:00 GMT

Abstract

Thanks to Mike for reviewing this.

This is a way to enable compile time customization of classes/functions in the style of C++ template meta-programming as in Modern C++ Design. In particular, we are going to implement the policy pattern, which is compile time version of the strategy pattern.

What do we gain by constraining ourselves to compile time customization, instead of run time one? High performance. Blazingly high performance. You gain an abstraction, without paying for it.

Too good to be true? Read on!

BTW1: none of this is new. It has been floating around in various forms. But I have never seen explained fully and associated to the policy pattern. BTW2: it is also related to the uber cool Shape proposal for C#, where it becomes an implementation detail.

Implementation

First, the usual plethora of namespaces … BenchmarkDotNet is really heavy into separating abstractions in different namespaces …

using System.Runtime.CompilerServices;
using System.Threading;

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Diagnosers;
using BenchmarkDotNet.Diagnostics.Windows.Configs;
using BenchmarkDotNet.Mathematics;
using BenchmarkDotNet.Order;
using BenchmarkDotNet.Running;

Let’s take a look at the strategy pattern, as more traditionally implemented. You got an interface IIncrementer and two implementations, either thread safe or not. Let’s also add a couple of struct implementations so that we can measure the performance difference of implementation by classes and by structs.

public interface IIncrementer { void Increment(ref int location); }

public sealed class CStandardIncrementer        : IIncrementer {[MethodImpl(MethodImplOptions.AggressiveInlining)] public void Increment(ref int location) => location += 1; }
public sealed class CInterlockedIncrementer     : IIncrementer {[MethodImpl(MethodImplOptions.AggressiveInlining)] public void Increment(ref int location) => Interlocked.Increment(ref location); }

public readonly struct SStandardIncrementer     : IIncrementer {[MethodImpl(MethodImplOptions.AggressiveInlining)] public void Increment(ref int location) => location += 1; }
public readonly struct SInterlockedIncrementer  : IIncrementer {[MethodImpl(MethodImplOptions.AggressiveInlining)] public void Increment(ref int location) => Interlocked.Increment(ref location); }

We then need a class that can be customized by an Incrementer. Think of it as: the policy of incrementing something is independent from the class in question.

Let’s take a Counter and call it Dynamic as we want to be able to customize it at runtime. We need to keep things simple so that looking at ASM is doable. Also we inline everything to try to squeeze the most performance out of our program.

public class DynamicCounter<T> where T: IIncrementer
{
    int _count;
    T _incrementer;

    public DynamicCounter(T incrementer) => _incrementer = incrementer;

    [MethodImpl(MethodImplOptions.AggressiveInlining)]
    public void Increment() => _incrementer.Increment(ref _count);
}

Then we look at how to implement the strategy pattern at compile time (transforming it magically into the policy pattern).

(no-one every talks about the negative aspects of giving names to things, one of these days I’ll blog about it …)

There are many ways to go about it. They all rely on the fact that the JIT Compiler instantiates a different type for each struct used to customize the type (aka each T).

For each one of these types, the JIT knows at compile type what T is, which brings about certain optimizations.

The optimization exploited by StaticCounterInterface is that the call to Increment becomes a non-virtual call that can be inlined.

public class StaticCounterInterface<T> where T : struct, IIncrementer
{
    int _count;
    T _incrementer = new T();

    [MethodImpl(MethodImplOptions.AggressiveInlining)]
    public void Increment() => _incrementer.Increment(ref _count);
}

For StaticCounterMatch the situation is even more obvious.

The Jitter doesn’t generate code for any of the if statements. It just puts the right code for the type T directly in the body of the Increment method.

It is as if the if statement were executed at compile time, as with C++ templates. Also notice the IncrementRaw method used for perf comparison.

public class StaticCounterMatch<T>
{
    int _count;

    [MethodImpl(MethodImplOptions.AggressiveInlining)]
    public void Increment() {
        if (typeof(T) == typeof(SStandardIncrementer))      _count += 1;
        if (typeof(T) == typeof(SInterlockedIncrementer))   Interlocked.Increment(ref _count);
    }

    [MethodImpl(MethodImplOptions.AggressiveInlining)]
    public void IncrementRaw() => _count += 1;
}

We are now ready for the performance testing. We create one class for each combination and benchmark their Increment method.

A few comments on the attributes: 1. DryCoreJob doesn’t do the benchmark, but just runs the diagnosers, in this case produces assembly code. 2. InProcessAttribute makes everything go faster, but cannot be used to generate assembly. 3. DisassemblyDiagnoser creates files with the assembly code. 4. RankColumn generates a nicer table.

//[DryCoreJob]
//[InProcessAttribute]
[DisassemblyDiagnoser(printAsm: true, printPrologAndEpilog: true, printIL: false, printSource: false, recursiveDepth: 3)]
[Orderer(SummaryOrderPolicy.FastestToSlowest)]
[RankColumn(NumeralSystem.Stars)]
public  class MainClass
{
    DynamicCounter<IIncrementer>         dynInterface           = new DynamicCounter<IIncrementer>(new CStandardIncrementer());
    DynamicCounter<CStandardIncrementer> dynConcrete            = new DynamicCounter<CStandardIncrementer>(new CStandardIncrementer());
    DynamicCounter<SStandardIncrementer> dynStruct              = new DynamicCounter<SStandardIncrementer>(new SStandardIncrementer());
    StaticCounterMatch<SStandardIncrementer> staticCounterM     = new StaticCounterMatch<SStandardIncrementer>();
    StaticCounterInterface<SStandardIncrementer> staticCounterI = new StaticCounterInterface<SStandardIncrementer>();

    [Benchmark] public void DynamicInterface()                      => dynInterface.Increment();
    [Benchmark] public void DynamicConcrete()                       => dynConcrete.Increment();
    [Benchmark] public void DynamicStruct()                         => dynStruct.Increment();
    [Benchmark] public void StaticCounterInterface()                => staticCounterI.Increment();
    [Benchmark] public void StaticCounterMatch()                    => staticCounterM.Increment();
    [Benchmark(Baseline = true)] public void IncrementRaw()         => staticCounterM.IncrementRaw();

    public static void Main() => BenchmarkRunner.Run<MainClass>();
}

Results

Please note that the results are valid just for the tested configuration.

I have no reason to think that they would be different on other modern runtimes/OSs as the optimizations are quite well known.

BenchmarkDotNet=v0.11.3, OS=Windows 10.0.17763.253 (1809/October2018Update/Redstone5)
Intel Core i7-6600U CPU 2.60GHz (Skylake), 1 CPU, 4 logical and 2 physical cores
.NET Core SDK=3.0.100-preview-009812
  [Host]     : .NET Core 2.0.9 (CoreCLR 4.6.26614.01, CoreFX 4.6.26614.01), 64bit RyuJIT
  DefaultJob : .NET Core 2.0.9 (CoreCLR 4.6.26614.01, CoreFX 4.6.26614.01), 64bit RyuJIT

Method	Mean	Error	StdDev	Median	Ratio	RatioSD	Rank
StaticCounterInterface	0.0000 ns	0.0000 ns	0.0000 ns	0.0000 ns	0.000	0.00	*
IncrementRaw	0.1036 ns	0.0515 ns	0.0573 ns	0.1071 ns	1.000	0.00	**
StaticCounterMatch	0.1122 ns	0.0422 ns	0.0943 ns	0.1020 ns	2.092	1.95	**
DynamicStruct	0.2707 ns	0.0910 ns	0.2683 ns	0.1407 ns	4.135	6.28	***
DynamicConcrete	1.9216 ns	0.1506 ns	0.4440 ns	1.7883 ns	23.417	21.44	****
DynamicInterface	2.2441 ns	0.1170 ns	0.3449 ns	2.1470 ns	32.783	30.52	*

As expected, you gain an order of magnitude in performance by foregoing run time customization, except when using a struct as the optimizer manages to inline that one (as we’ll see).

Notice that these numbers are really low. In fact the order of the first 4 lines might change when you run it. But they are always much faster than the rest.

But why? Let’s look at the generated code.

IL and ASM

First let’s look at IL for a few of the methods

MainClass.IncrementRaw()
       IL_0000: ldarg.0
       IL_0001: ldfld StaticCounterMatch`1<SStandardIncrementer> MainClass::staticCounterM
       IL_0006: callvirt System.Void StaticCounterMatch`1<SStandardIncrementer>::IncrementRaw()
       IL_000b: ret

; MainClass.StaticCounterInterface()
       IL_0000: ldarg.0
       IL_0001: ldfld StaticCounterInterface`1<SStandardIncrementer> MainClass::staticCounterI
       IL_0006: callvirt System.Void StaticCounterInterface`1<SStandardIncrementer>::Increment()
       IL_000b: ret

; MainClass.StaticCounterMatch()
       IL_0000: ldarg.0
       IL_0001: ldfld StaticCounterMatch`1<SStandardIncrementer> MainClass::staticCounterM
       IL_0006: callvirt System.Void StaticCounterMatch`1<SStandardIncrementer>::Increment()
       IL_000b: ret

Ok, nothing interesting there apart from the use of callvirt when you would think a standard call would do (i.e. IncrementRaw).

I vaguely remember from my C# compiler days that we do that as a way to short-circuit the test for null, as callvirt does it automatically.

The assembly code is more interesting. Let’s start from the three fast-resolved-at-compile-time methods.

BTW: remember that looking at optimized ASM code is like peering into a muddy lake with foggy glasses. Let’s do it.

; MainClass.IncrementRaw()
       mov     rax,qword ptr [rcx+20h]
       inc     dword ptr [rax+8]
       ret

; MainClass.StaticCounterInterface()
       mov     rax,qword ptr [rcx+28h]
       (*)mov     edx,dword ptr [rax]
       add     rax,8
       inc     dword ptr [rax]
       ret

; MainClass.StaticCounterMatch()
       mov     rax,qword ptr [rcx+20h]
       (*)mov     edx,dword ptr [rax]
       inc     dword ptr [rax+8]
       ret

; MainClass.DynamicStruct()
       mov     rax,qword ptr [rcx+18h]
       (*) mov     edx,dword ptr [rax]
       add     rax,8
       inc     dword ptr [rax]
       ret

Yep, they are all the same (apart from the mysterious * instruction)! Get the memory location of the field, increment it.

The Jitter has completely inlined the code. It is as if you had written the incrementing code directly into the function, despite you composing the type using two independent abstractions.

I think that is pretty cool! You can abstract your code properly and not pay the price for it (well, apart from one assembly instruction).

For the sake of completeness, let’s look at the assembly code for the dynamic dispatching cases.

; MainClass.DynamicConcrete()
       mov     rcx,qword ptr [rcx+10h]
       cmp     dword ptr [rcx],ecx
       lea     rdx,[rcx+10h]
       mov     rcx,qword ptr [rcx+8]
       mov     r11,7FF8C2B30470h
       mov     rax,qword ptr [r11]
       cmp     dword ptr [rcx],ecx
       jmp     rax

; MainClass.DynamicInterface()
       mov     rcx,qword ptr [rcx+8]
       cmp     dword ptr [rcx],ecx
       lea     rdx,[rcx+10h]
       mov     rcx,qword ptr [rcx+8]
       mov     r11,7FF8C2B10470h
       mov     rax,qword ptr [r11]
       cmp     dword ptr [rcx],ecx
       jmp     rax

The first thing to notice is that the code is identical, despite their seemingly different declarations. The Jitter doesn’t care.

Notice the machinations the Jitter performs, very likely related to dynamic dispatching, to calculate the address to finally jump to. That’s where our Increment method is located.

No wonder it is slower.

Summary

If you can afford to use the policy pattern instead of the more generic strategy pattern (i.e. compile time vs run time dispatch) and/or you need bare to the metal performance, consider the code above.

As for me, I plan to use it in the near future for a few low level abstractions (i.e. memory allocators).

Stopping Garbage Collection in .NET Core 3.0 (part II)

Mon, 21 Jan 2019 10:05:45 GMT

Let’s see how it’s implemented. For why it is implemented, see part I.

Thanks to Mike for reviewing this.

using System;
using System.Diagnostics.Tracing;
using System.Runtime;

The FxCop code analyzers get upset if I don’t declare this, which also impede me from using unsigned numeral types in interfaces.

[assembly: CLSCompliant(true)]

namespace LNativeMemory
{

The first piece of the puzzle is to implement an event listener. It is a not-obvious (for me) class. I don’t fully understand the lifetime semantics, but the code below seems to do the right thing.

The interesting piece is _started and the method Start(). The constructor for EventListener allocates plenty of stuff. I don’t want to do those allocations after calling TryStartNoGCRegion because they would use part of the GC Heap that I want for my program. Instead, I create it before such call, but then I make it ‘switch on’ just after the Start() method is called.

    internal sealed class GcEventListener : EventListener
    {
        Action _action;
        EventSource _eventSource;
        bool _active = false;

        internal void Start() { _active = true; }
        internal void Stop() { _active = false; }

As described in part one, you pass a delegate at creation time, which is called when garbage collection is restarted.

        internal GcEventListener(Action action) => _action = action ?? throw new ArgumentNullException(nameof(action));

We register to all the events coming from .NET. We want to call the delegate at the exact point when garbage collection is turned on again. We don’t have a clean way to do that (aka there is no runtime event we can hook up to, see here, so listening to every single GC event gives us the most chances of doing it right. Also it ties us the least to any pattern of events, which might change in the future.

        // from https://docs.microsoft.com/en-us/dotnet/framework/performance/garbage-collection-etw-events
        private const int GC_KEYWORD = 0x0000001;
        private const int TYPE_KEYWORD = 0x0080000;
        private const int GCHEAPANDTYPENAMES_KEYWORD = 0x1000000;

        protected override void OnEventSourceCreated(EventSource eventSource)
        {
            if (eventSource.Name.Equals("Microsoft-Windows-DotNETRuntime", StringComparison.Ordinal))
            {
                _eventSource = eventSource;
                EnableEvents(eventSource, EventLevel.Verbose, (EventKeywords)(GC_KEYWORD | GCHEAPANDTYPENAMES_KEYWORD | TYPE_KEYWORD));
            }
        }

For each event, I check if the garbage collector has exited the NoGC region. If it has, then let’s invoke the delegate.

        protected override void OnEventWritten(EventWrittenEventArgs eventData)
        {
            var eventName = eventData.EventName;
            if(_active && GCSettings.LatencyMode != GCLatencyMode.NoGCRegion)
            {
                _action?.Invoke();
            }
        }
    }

Now that we have our event listener, we need to hook it up. The code below implements what I described earlier.

Do your allocations for the event listener
Start the NoGc region
Start monitoring the runtime for the start of the NoGC region

    public static class GC2
    {
        static private GcEventListener _evListener;

        public static bool TryStartNoGCRegion(long totalSize, Action actionWhenAllocatedMore)
        {

            _evListener = new GcEventListener(actionWhenAllocatedMore);
            var succeeded = GC.TryStartNoGCRegion(totalSize, disallowFullBlockingGC: false);
            _evListener.Start();

            return succeeded;
        }

As puzzling as this might be, I provisionally believe it to be correct. Apparently, even if the GC is not in a NoGC region, you still need to call EndNoGCRegion if you have called TryStartNoGCRegion earlier, otherwise your next call to TryStartNoGCRegion will fail. EndNoGCRegion will throw an exception, but that’s OK. Your next call to TryStartNoGCRegion will now succeed.

Now read the above repeatedly until you got. Or just trust that it works somehow.

        public static void EndNoGCRegion()
        {
            _evListener.Stop();

            try
            {
                GC.EndNoGCRegion();
            } catch (Exception)
            {

            }
        }
    }

This is used as the default behavior for the delegate in the wrapper class below. I was made aware by the code analyzer that I shouldn’t be throwing an OOF exception here. At first, I dismissed it, but then it hit me. It is right.

We are not running out of memory here. We simply have allocated more memory than what we declared we would. There is likely plenty of memory left on the machine. Thinking more about it, I grew ashamed of my initial reaction. Think about a support engineer getting an OOM exception at that point and trying to figure out why. So, always listen to Lint …

    public class OutOfGCHeapMemoryException : OutOfMemoryException {
        public OutOfGCHeapMemoryException(string message) : base(message) { }
        public OutOfGCHeapMemoryException(string message, Exception innerException) : base(message, innerException) { }
        public OutOfGCHeapMemoryException() : base() { }

    }

This is an utility class that implements the IDisposable pattern for this scenario. The size of the default ephemeral segment comes from here.

    public sealed class NoGCRegion: IDisposable
    {
        static readonly Action defaultErrorF = () => throw new OutOfGCHeapMemoryException();
        const int safeEphemeralSegment = 16 * 1024 * 1024;

        public NoGCRegion(int totalSize, Action actionWhenAllocatedMore)
        {
            var succeeded = GC2.TryStartNoGCRegion(totalSize, actionWhenAllocatedMore);
            if (!succeeded)
                throw new InvalidOperationException("Cannot enter NoGCRegion");
        }

        public NoGCRegion(int totalSize) : this(totalSize, defaultErrorF) { }
        public NoGCRegion() : this(safeEphemeralSegment, defaultErrorF) { }

        public void Dispose() => GC2.EndNoGCRegion();
    }
}

Stopping Garbage Collection in .NET Core 3.0 (part I)

Mon, 14 Jan 2019 12:34:16 GMT

Scenario

Thanks to Mike for reviewing this.

You have an application or a particular code path of your application that cannot take the pauses that GC creates. Typical examples are real time systems, tick by tick financial apps, embedded systems, etc …

Disclaimer

For any normal kind of applications, YOU DON’T NEED TO DO THIS. You are likely to make your application run slower or blow up memory. If you have an hot path in your application (i.e. you are creating an editor with Intellisense), use the GC latency modes. Use the code below just under extreme circumstance as it is untested, error prone and wacky. You are probably better off waiting for an official way of doing it (i.e. when this is implemented)

The problem with TryStartNoGCRegion

There is a GC.TryStartNoGCRegion in .NET. You can use it to stop garbage collection passing a totalBytes parameter that represents the maximum amount of memory that you plan to allocate from the managed heap. Matt describes it here.

The problem is that when/if you allocate more than that, garbage collection resumes silently. Your application continues to work, but with different performance characteristics from what you expected.

The idea

The main idea is to use ETW events to detect when a GC occurs and to call an user provided delegate at that point. You can then do whatever you want in the delegate (i.e. shutdown the process, send email to support, start another NoGC region, etc…).

Also, I have wrapped the whole StartNoGCRegion/EndNoGCRegion in an IDisposable wrapper for easy of use.

The tests

Let’s start by looking at how you use it.

using Xunit;
using System.Threading;

namespace LNativeMemory.Tests
{

    // XUnit executes all tests in a class sequentially, so no problem with multi-threading calls to GC
    public class GC2Tests
    {

We need to use a timer to maximize the chances that a GC happens in some of the tests. Also we allocate an amount that should work in all GC configuration as per the article above. trigger is a static field so as to stay zero-allocation (otherwise the delegate will have to capture the a local trigger variable creating a heap allocated closure). Not that it matters any to be zero-allocation in this test, but I like to keep ClrHeapAllocationAnalyzer happy.

BTW: XUnit executes all tests in a class sequentially, so no problem with multi-threading calls to GC.

        const int sleepTime = 200;
        const int totalBytes = 16 * 1024 * 1024;
        static bool triggered = false;

First we test that any allocation that doesn’t exceed the limit doesn’t trigger the call to action.

        [Fact]
        public void NoAllocationBeforeLimit()
        {
            try
            {
                triggered = false;
                var succeeded = GC2.TryStartNoGCRegion(totalBytes, () => triggered = true);
                Assert.True(succeeded);
                Thread.Sleep(sleepTime);
                Assert.False(triggered);

                var bytes = new byte[99];
                Thread.Sleep(sleepTime);
                Assert.False(triggered);
            }
            finally
            {
                GC2.EndNoGCRegion();
                triggered = false;
            }
        }

Then we test that allocating over the limit does trigger the action. To do so we need to trigger a garbage collection. Out best attempt is with the goofy for loop. If you got a better idea, shout.

        [Fact]
        public void AllocatingOverLimitTriggersTheAction()
        {
            try
            {
                triggered = false;
                var succeeded = GC2.TryStartNoGCRegion(totalBytes, () => triggered = true);
                Assert.True(succeeded);
                Assert.False(triggered);

                for (var i = 0; i < 3; i++) { var k = new byte[totalBytes]; }

                Thread.Sleep(sleepTime);
                Assert.True(triggered);
            }
            finally
            {
                GC2.EndNoGCRegion();
                triggered = false;
            }
        }

We also test that we can go back and forth between starting and stopping without messing things up.

        [Fact]
        public void CanCallMultipleTimes()
        {

            for (int i = 0; i < 3; i++)
            {
                NoAllocationBeforeLimit();
            }
        }

And lastly, we make sure that we can use our little wrapper function, just to be sure everything works.

        [Fact]
        public void CanUseNoGCRegion()
        {
            triggered = false;
            using (new NoGCRegion(totalBytes, () => triggered = true))
            {
                for (var i = 0; i < 3; i++) { var k = new byte[totalBytes]; }
                Thread.Sleep(sleepTime);
                Assert.True(triggered);
                triggered = false;
            }
        }
    }
}

A Stack data structure implementation using Span

Mon, 04 Jun 2018 11:56:54 GMT

I am back in Microsoft and today we talk about the code below, which is on github here:

public ref struct SpanStack<T>
{
    private Span memory;
    private int index;
    private int size;

    public SpanStack(Span mem) { memory = mem; index = 0; size = mem.Length; }

    public bool IsEmpty() => index < 0;
    public bool IsFull() => index > size - 1;

    public void Push(T item) => memory[index++] = item;
    public T Pop() => memory[--index];
}

public static class SpanExtensions
{
    public static SpanStack AsStack<T>(this Span span) => new SpanStack(span);
}

This Stack data structure can be used over memory that resides on the stack, heap or unmanaged heap. If you know about Span this should immediately make sense to you.

This has to be a ref struct because it contains a Span. It can’t be used on the heap (i.e. in lambdas, async, class field, …). You have to build it on top of Memory if you need that. Also, you can happily blow the stack with this guy …

Let’s micro-benchmark it with BenchmarkDotNet. For example, a postfix calculator. Let’s first do it naively using inheritance and the generic Stack class in the framework.

This is the naive object model:

abstract class Token {}

sealed class Operand: Token
{
    public int Value { get; }
    public Operand(int v) { Value = v; }
}

abstract class Operator: Token {
    abstract public int Calc(int a, int b);
}

sealed class Add: Operator
{
    public override int Calc(int a, int b) => a + b;
}

sealed class Mult : Operator
{
    public override int Calc(int a, int b) => a * b;
}

sealed class Minus : Operator
{
    public override int Calc(int a, int b) => a - b;
}
Let's then do it trying to be a bit more performance aware using a stack friendly representation:

public enum TokenType { Operand, Sum, Mult, Minus}

readonly struct SToken
{
    public TokenType Type { get; }
    public int Value { get; }
    public SToken(TokenType t, int v) { Type = t; Value = v; }
    public SToken(TokenType t) { Type = t; Value = 0; }

    public int Calc(int a, int b) =>
               Type == TokenType.Sum   ? a + b :
               Type == TokenType.Minus ? a - b :
               Type == TokenType.Minus ? a * b :
               throw new Exception("I don't know that one");

}

Perhaps not overtly elegant, but not that terrible either. You got to love those expression bodied methods and throw-expression.

We then setup things (I know I could/should parse a string here):

static Token[] tokens;
static SToken[] stokens;

[GlobalSetup]
public void Setup()
{
    tokens = new Token[] { new Operand(2), new Operand(3), new Operand(4), new Add(),
                           new Mult(), new Operand(5), new Minus() };
    stokens = new SToken[] { new SToken(TokenType.Operand, 2),
                             new SToken(TokenType.Operand, 3), new SToken(TokenType.Operand, 4),
                             new SToken(TokenType.Sum),  new SToken(TokenType.Mult),
                             new SToken(TokenType.Operand, 5), new SToken(TokenType.Minus)};
}

And first test the naive object model with the standard Stack from System.Collections.Generic.

[Benchmark]
public int PostfixEvalStack()
{
    var stack = new Stack(100);

    foreach (var token in tokens)
    {
        switch (token)
        {
            case Operand t:
                stack.Push(t);
                break;
            case Operator o:
                var a = stack.Pop() as Operand;
                var b = stack.Pop() as Operand;
                var result = o.Calc(a.Value, b.Value);
                stack.Push(new Operand(result));
                break;
        }
    }
    return (stack.Pop() as Operand).Value;
}

Then let’s just swap out our own lean-and-mean stack:

[Benchmark]
public int PostfixEvalSpanStack()
{
    Span span = new Token[100];
    var stack = span.AsStack();

    foreach (var token in tokens)
    {
        switch (token)
        {
            case Operand t:
                stack.Push(t);
                break;
            case Operator o:
                var a = stack.Pop() as Operand;
                var b = stack.Pop() as Operand;
                var result = o.Calc(a.Value, b.Value);
                stack.Push(new Operand(result));
                break;
        }
    }
    return (stack.Pop() as Operand).Value;
}

And finally let’s go the whole way, lean object model and lean data structure, everything on the stack:

[Benchmark(Baseline = true)]
public int PostfixEvalSpanStackStructTypes()
{
    Span span = stackalloc SToken[100];
    var stack = span.AsStack();

    foreach (var token in stokens)
    {
        if (token.Type == TokenType.Operand)
        {
            stack.Push(token);
        } else {
            var a = stack.Pop();
            var b = stack.Pop();
            var result = token.Calc(a.Value, b.Value);
            stack.Push(new SToken(TokenType.Operand, result));
            break;
        }
    }
    return stack.Pop().Value;
}

We also want to check that we didn’t code anything stupid and finally run the benchmark.

static void Test()
{
    var p = new Program();
    p.Setup();
    Trace.Assert(p.PostfixEvalStack() == p.PostfixEvalSpanStack() &&
                 p.PostfixEvalSpanStack() == p.PostfixEvalSpanStackStructTypes());
}
static void Main(string[] args)
{
    Test();
    var summary = BenchmarkRunner.Run();
}

On my machine I get these results:

BenchmarkDotNet=v0.10.14, OS=Windows 10.0.16299.431 (1709/FallCreatorsUpdate/Redstone3)
Intel Core i7-6600U CPU 2.60GHz (Skylake), 1 CPU, 4 logical and 2 physical cores
Frequency=2742185 Hz, Resolution=364.6727 ns, Timer=TSC
.NET Core SDK=2.1.300-rc1-008673
  [Host]     : .NET Core 2.1.0-rc1 (CoreCLR 4.6.26426.02, CoreFX 4.6.26426.04), 64bit RyuJIT
  DefaultJob : .NET Core 2.1.0-rc1 (CoreCLR 4.6.26426.02, CoreFX 4.6.26426.04), 64bit RyuJIT
Method  Mean    Error   StdDev  Scaled  ScaledSD
PostfixEvalSpanStackStructTypes 76.24 ns    1.563 ns    2.857 ns    1.00    0.00
PostfixEvalSpanStack    168.65 ns   5.280 ns    15.319 ns   2.22    0.22
PostfixEvalStack    334.56 ns   7.387 ns    20.593 ns   4.39    0.31

Your mileage might vary. I want to emphasize that I am just playing with things. I haven’t done any deep analysis of this benchmark. There can be flaws, etc… etc…

Still, I find the idea of data structures which are memory-location-independent rather fascinating.

Building a stock alert system with Google Script

Fri, 31 Oct 2014 08:11:13 GMT

This is obsolete as Yahoo stopped their quote service. I have a new spreadsheet here.

When I thought about it, I realized that my ideal system would be a spreadsheet where to add tickers and alert levels. Under the covers, the system would need to check the current price of a ticker, compare it with the alert level and send me email when triggered.

Also the whole thing shouldn’t be running from my machine at home, but from somewhere on the internet.

Google script fit the bill. Let’s see how it works.

Script is here. Sheet is here.

First a utility function to send errors via email, which will be used throughout the script.

function emailError(e) {
  MailApp.sendEmail("lucabolg@gmail.com", "Watchlist Error",
                    "\r\nMessage: " + e.message
                    + "\r\nFile: " + e.fileName
                    + "\r\nLine: " + e.lineNumber
                    + "\r\nLOg: " + Logger.getLog())
}

Then another one to check if the price downloaded from the internet is sensible.

function validPrice(price) {
 return price != 'undefined' && price > 0.1
}

We then need one to retrieve the current price of a ticker from the array of data returned from the internet:

// Find the current price of a ticker in an array of data where the ticker is the first column
function getQuote(data, ticker) {
  var ticker = ticker.trim().toLowerCase()

  for(var i = 0; i = 22 && hour <= 23) || value != "close"
}

With all of that in place, we can now look at the main function. First we load up the spreadsheet and get the values and headers we care about. This would be more robust if we looked up the sheet by name. Also the id of the sheet is burned in the code. You’ll need to change it if you want to make it point to your own.

// Check spreadsheet with tickers and stop prices, send email when a stop is hit and mark the row as 'Executed'.
function checkQuotes() {
 try {

 // Get all data from spreadsheet in one web call.
 var ss = SpreadsheetApp.openById("1WQf2AiBPQW5HLzCyGgsFlKN0f1HTOWAteJ5bJCXVnlc")
 var range = ss.getSheets()[0].getDataRange()
 var values = range.getValues()
 var headers = values[0]
 var rows = ObjApp.rangeToObjects(values)
 var body = ""
 var now = new Date()

Notice ‘ObjApp’ is part of the ObjService library to make the code a bit more maintainable, instead of scattering column numbers in the code.

Now we get all the tickers and download the prices from Yahoo (we try three times as it occasionally fails.

    // Fish out all tickers from col 0 where Status (col 4) is not executed
    var tickers = []
    for(var i = 1; i < rows.length; i++) {// dont' process the headers
      if((rows[i]).executed.toLowerCase() == 'active' && isRightTime(rows[i], now)) tickers.push((rows[i]).ticker.trim().toLowerCase())
    }
    Logger.log("Tickers:%s" ,tickers)

    if(tickers.length == 0) return // Nothing to process

    // Get ticker, real time bid, real time ask for all tickers in one web call
    var url = "http://finance.yahoo.com/d/quotes.csv?s=" + tickers.join("+") + "&f=sl1"//"&f=sb2b3"

    // Try 3 times before giving up
    for(var i = 0; i < 3; i++) {
      try {
        var response = UrlFetchApp.fetch(url)
        break;
      } catch(e) {
      }
    }

    Logger.log("Response:\n%s", response)
    var data = Utilities.parseCsv(response.getContentText())
    Logger.log("Data:\n%s", data)

Once that is done, we enter the main loop. The concept is simple, for each row we check the price and, if the price is above/below the alert we add it to the body string and mark the row in the sheet so that we don’t process it again next time. A the end, we email the body variable if not null.

First we check that we haven’t already executed this row:

    for(var i = 1; i < rows.length; i++) {// dont' process the headers
      var current = rows[i]
      if(current.executed.trim().toLowerCase() == 'executed') continue // no need to process it as it is 'Executed'

      var symbol = current.operator
      var stop = current.stop

If it’s still active and if it is the right time, we check if the alert is triggered. If it is we add the text to the body variable.

      if(isRightTime(current, now)) {
        var price = getQuote(data, current.ticker)
        if( (symbol.trim() == ">" && price > stop) ||
           (symbol.trim() == "<" && price < stop)) {

          current.executed = "Executed"
          current.price = price

          body += [current.kind, current.ticker, current.price, current.operator, current.stop, "\r\n"].join(" ")
          Logger.log("Body in loop:\n%s", body)
        }
      }
    }

If body is not empty, that means that something was triggered, so we send the email.

    if(body != "") {
      Logger.log("Body final:%s", body)
      MailApp.sendEmail('lucabolg@gmail.com', 'Watchlist: stops triggered', body)
      var data = ObjApp.objectToArray(headers, rows)
      data.unshift(headers)
      range.setValues(data)
    }

If an error was generated, then we send the error email.

  } catch (e) {
    Logger.log(e.lineNumber + ":" + e.message)
    emailError(e)
  }
}

My experience overall was remarkable. The learning curve was very quick and the web editor works remarkably well (well, stepping through code is rather slow).

Overall, if Google has all your data (in Drive) and you can write code to manipulate it (in Google script), why do I need my home computer again? I can just have a small screen that connects to the internet and I’m done.

That’s probably true for me apart from two things that I haven’t found in web form: editing of images in the raw format and a sophisticated portfolio application. If I find these two, I’m ready to give up my life to Google …

Funky C for literate programming

Wed, 20 Mar 2013 15:29:05 GMT

Main ideas

This is a port of LLIte in C. The reason for it is to experiment with writing functional code in standard C and compare the experience with using a functional language like F#. It is in a way a continuation of my previous posts on the topic.

I will be using glib and an header of convenient macros/functions to help me (lutils.h). I don’t think that is cheating. Any modern C praticoner has its bag of tricks …

Don’t tell me this is not idiomatic C. I already know that.

#include <string.h>
#include <stdbool.h>

#include <glib.h>
#include <glib/gprintf.h>

#ifdef ARENA
#include "arena.h"
#endif

#include "lutils.h"

Lack of tuples

In the snippet below I overcomed such deficiency by declaring a struct. Using the new constructor syntax makes initializing a static table simple.

typedef struct LangSymbols { char language[40]; char start[10]; char end[10];} LangSymbols;

static
LangSymbols* s_lang_params_table[] = {
    &(LangSymbols) {.language = "fsharp",   .start = "(*" "*", .end = "*" "*)"},
    &(LangSymbols) {.language = "c",        .start = "/*" "*", .end = "*" "*/"},
    &(LangSymbols) {.language = "csharp",   .start = "/*" "*", .end = "*" "*/"},
    &(LangSymbols) {.language = "java",     .start = "/*" "*", .end = "*" "*/"},
    NULL
};

Folding over arrays

I need to gather all the languages, aka perform a fold over the array. You might have noticed the propensity to add a NULL terminator marker to arrays (as for strings). This allows me to avoid passing a size to functions and makes simpler writing utility macros (as foreach below) more simply.

In the rest of the program, every time I end a function with _z, it is because I consider it generally usable and I add a version of it without the _z to lutils.h.

#define array_foreach_z(p) for(; *symbols != NULL; ++symbols)

static
char* summary(LangSymbols** symbols) {

    GString* langs = g_string_sized_new(20);
    array_foreach(symbols) g_string_append_printf(langs, "%s ", (*symbols)->language);

    g_string_truncate(langs, strlen(langs->str) - 1);

    GString* usage = g_string_sized_new(100);

    g_string_printf(usage,
        "You should specify:nt. either -l or -o and -pn"
        "t. either -indent or -P and -Cn"
        "t. -l supports: %s"
        ,langs->str);

    return usage->str;
}

Find an item in an array based on some expression. Returns NULL if not found. Again, this is a common task, hence I’ll abstract it out with a macro (that ends up being a cute use of gcc statment expressions).

#define array_find_z(arr, ...)                          
    ({                                                  
        array_foreach(arr) if (__VA_ARGS__) break;      
        *arr;                                           
    })

static
LangSymbols* lang_find_symbols(LangSymbols** symbols, char* lang) {
    g_assert(symbols);
    g_assert(lang);

    return array_find(symbols, !strcmp((*symbols)->language, lang));
}

Deallocating stuff

You might wonder why I don’t seem overly worried about deallocating the memory that I allocate. I haven’t gone crazy(yet). You’ll see.

Discriminated unions

Here are the discriminated unions macros from a previous blog post of mine. I’ll need a couple of these and pre-declare two functions.

union_decl(CodeSymbols, Indented, Surrounded)
    union_type(Indented,    int indentation;)
    union_type(Surrounded,  char* start_code; char* end_code;)
union_end(CodeSymbols);

typedef struct Options {
    char*           start_narrative;
    char*           end_narrative;
    CodeSymbols*    code_symbols;
} Options;

static
gchar* translate(Options*, gchar*);

union_decl(Block, Code, Narrative)
    union_type(Code,        char* code)
    union_type(Narrative,   char* narrative)
union_end(Block);

Main data structure

We want to use higher level abstractions that standard C arrays, hence we’ll pick a convenient data structure to use in the rest of the code. A queue lets you to insert at the front and back, with just a one pointer overhead over a single linked list. Hence it is my data structure of choice for this program.

static
GQueue* blockize(Options*, char*);
There is already a function in glib to check if a string has a certain prefix (g_str_has_prefix). We need one that returns the remaining string after the prefix. We also define a g_slow_assert that is executed just if G_ENABLE_SLOW_ASSERT is defined

static
char* str_after_prefix(char* src, char* prefix) {
    g_assert(src);
    g_assert(prefix);
    g_slow_assert(g_str_has_prefix(src, prefix));

    while(*prefix != '0')
        if(*src == *prefix) ++src, ++prefix;
        else break;

    return src;
}

Tokenizer

The structure of the function is identical to the F# version. The big bread-winners are statement expressions and local functions …

It is interesting how you can replicate the shape of an F# function by substituting ternary operators for match statements.

It is nothing magic, just a way to have a case statment as an expression, but it is suggestive of its more functional counterpart.

#define NL "n"

union_decl(Token, OpenComment, CloseComment, Text)
    union_type(OpenComment, int line)
    union_type(CloseComment,int line)
    union_type(Text,        char* text)
union_end(Token);

GQueue* tokenize(Options* options, char* source) {
    g_assert(options);
    g_assert(source);

    struct tuple { int line; GString* acc; char* rem;};

    bool is_opening(char* src)      { return g_str_has_prefix(src, options->start_narrative);}
    bool is_closing(char* src)      { return g_str_has_prefix(src, options->end_narrative);}
    char* remaining_open (char* src){ return str_after_prefix(src, options->start_narrative);}
    char* remaining_close(char* src){ return str_after_prefix(src, options->end_narrative);}

    struct tuple text(char* src, GString* acc, int line) {
        inline struct tuple stop_parse_text()
            { return (struct tuple) {.line = line, .acc = acc, .rem = src};}

        return  str_empty (src)? stop_parse_text() :
                is_opening(src)? stop_parse_text() :
                is_closing(src)? stop_parse_text() :
                                ({
                                  int line2         = g_str_has_prefix(src, NL) ? line + 1
                                                                                : line;
                                  GString* newAcc   = g_string_append_c(acc, *src);
                                  char* rem         = src + 1;
                                  text(rem, newAcc, line2);
                                });
    }

    GQueue* tokenize_rec(char* src, GQueue* acc, int line) {
        return  str_empty(src)  ?   acc                     :
                is_opening(src) ?   tokenize_rec(remaining_open(src),
                                        g_queue_push_back(acc, union_new(
                                                    Token, OpenComment, .line = line)),
                                        line)        :
                is_closing(src) ?   tokenize_rec(remaining_close(src),
                                               g_queue_push_back(acc, union_new(
                                                    Token, CloseComment, .line = line)),
                                        line)        :
                                ({
                                    struct tuple t = text(src, g_string_sized_new(200), line);
                                    tokenize_rec(t.rem,
                                        g_queue_push_back(acc, union_new(
                                                    Token, Text, .text = t.acc->str)), t.line);
                                 });
    }

    return tokenize_rec(source, g_queue_new(), 1);
}

Parser

This again has a similar structure to the F# version, just longer. It is very long because it contains 3 (nested) functions which are on the verbose side in C.

The creation of a error macro is unfortunate. I just don’t know how to adapt g_assert_e so that it works for not pointer returning functions.

I also need a simple function report_error to exit gracefully giving a message to the user. I didn’t found such thing in glib (?)

#define report_error_z(...) G_STMT_START { g_print(__VA_ARGS__); exit(1); } G_STMT_END                                                            

union_decl(Chunk, NarrativeChunk, CodeChunk)
    union_type(NarrativeChunk,  GQueue* tokens)
    union_type(CodeChunk,       GQueue* tokens)
union_end(Chunk);

static
GQueue* parse(Options* options, GQueue* tokens) {
    g_assert(options);
    g_assert(tokens);

    struct tuple { GQueue* acc; GQueue* rem;};

    #define error(...) 
        ({ report_error(__VA_ARGS__); (struct tuple) {.acc = NULL, .rem = NULL}; })

    struct tuple parse_narrative(GQueue* acc, GQueue* rem) {

        bool isEmpty    = g_queue_is_empty(rem);
        Token* h        = g_queue_pop_head(rem);
        GQueue* t       = rem;

        return  isEmpty                 ?
                                    error("You haven't closed your last narrative comment") :
                h->kind == OpenComment  ?
                    error("Don't open narrative comments inside narrative comments at line %i",
                          h->OpenComment.line)                                              :
                h->kind == CloseComment ? (struct tuple) {.acc = acc, .rem = t}             :
                h->kind == Text         ? parse_narrative(g_queue_push_back(acc, h), t)     :
                                          error("Should never get here");
    };

    struct tuple parse_code(GQueue* acc, GQueue* rem) {

        bool isEmpty    = g_queue_is_empty(rem);
        Token* h    = g_queue_pop_head(rem);
        GQueue* t   = rem;

        return  isEmpty                 ? (struct tuple) {.acc = acc, .rem = t}         :
                h->kind == OpenComment  ?
                    (struct tuple) {.acc = acc, .rem = g_queue_push_front(rem, h)}      :
                h->kind == CloseComment ? parse_code(g_queue_push_back(acc, h), rem)    :
                h->kind == Text         ? parse_code(g_queue_push_back(acc, h), rem)    :
                                          error("Should never get here");
    };
    #undef error

    GQueue* parse_rec(GQueue* acc, GQueue* rem) {

        bool isEmpty    = g_queue_is_empty(rem);
        Token* h    = g_queue_pop_head(rem);
        GQueue* t   = rem;

        return  isEmpty                 ? acc                                           :
                h->kind == OpenComment  ? ({
                                           GQueue* emp = g_queue_new();
                                           struct tuple tu = parse_narrative(emp, t);
                                           Chunk* ch = union_new(
                                                Chunk, NarrativeChunk, .tokens = tu.acc );
                                           GQueue* newQ = g_queue_push_back(acc, ch);
                                           parse_rec(newQ, tu.rem);
                                           })                                            :
                h->kind == CloseComment ?
                    report_error_e(
                        "Don't insert a close narrative comment at the start of your"
                        " program at line %i",
                                            h->OpenComment.line)                         :
                h->kind == Text         ?
                                        ({
                                           GQueue* emp = g_queue_new();
                                           struct tuple tu =
                                                parse_code(g_queue_push_front(emp, h), t);
                                           parse_rec(g_queue_push_back
                                            (acc,
                                             union_new(Chunk, CodeChunk, .tokens = tu.acc)),
                                             tu.rem);
                                          })                                                               :
                                          g_assert_no_match;
    }

    return parse_rec(g_queue_new(), tokens);
}

Flattener

This follows the usual practice of representing fold as foreach statments (and maps to). Pheraps I shall build better abstractions for them at some point. I also introduce a little macro to simplify writing of GFunc lambdas, given how pervasive they are.

Again, note how heavy ternary operated this is …

#define g_func_z(type, name, ...) lambda(void,                                              
                                        (void* private_it, G_GNUC_UNUSED void* private_no){ 
                                       type name = private_it;                              
                                       __VA_ARGS__                                          
                                })

static
GQueue* flatten(Options* options, GQueue* chunks) {
    GString* token_to_string_narrative(Token* tok) {
        return  tok->kind == OpenComment ||
                tok->kind == CloseComment   ?
                    report_error_e("Cannot nest narrative comments at line %i",
                                   tok->OpenComment.line)                                   :
                tok->kind == Text           ? g_string_new(tok->Text.text)                  :
                                              g_assert_no_match;
    }
    GString* token_to_string_code(Token* tok) {
        return  tok->kind == OpenComment    ?
                report_error_e(
                    "Open narrative comment cannot be in code at line %i."
                    " Pheraps you have an open comment "
                    "in a code string before this comment tag?"
                    , tok->OpenComment.line)                                                :
                tok->kind == CloseComment   ? g_string_new(options->end_narrative)          :
                tok->kind == Text           ? g_string_new(tok->Text.text)                  :
                                              g_assert_no_match;
    }
    Block* flatten_chunk(Chunk* ch) {
        return  ch->kind == NarrativeChunk  ? ({
                               GQueue* tokens = ch->NarrativeChunk.tokens;
                               GString* res = g_string_sized_new(256);
                               g_queue_foreach(tokens, g_func(Token*, tok,
                                                g_string_append(
                                                    res,
                                                    token_to_string_narrative(tok)->str);
                                                ), NULL);
                               union_new(Block, Narrative, .narrative = res->str);
                                               })   :
                ch->kind == CodeChunk       ? ({
                               GQueue* tokens = ch->CodeChunk.tokens;
                               GString* res = g_string_sized_new(256);
                               g_queue_foreach(tokens, g_func(Token*, tok,
                                                        g_string_append(
                                                            res,
                                                            token_to_string_code(tok)->str);
                                                        ), NULL);
                               union_new(Block, Code, .code = res->str);
                                               })   :
                               g_assert_no_match;
    }

    GQueue* res = g_queue_new();
    g_queue_foreach(chunks, g_func(Chunk*, ch,
                                Block* b = flatten_chunk(ch);
                                g_queue_push_tail(res, b);
                                ) ,NULL);
    return res;
}

Now we can tie everything together to build blockize, which is our parse tree.

static
GQueue* blockize(Options* options, char* source) {
    GQueue* tokens  = tokenize(options, source);
    GQueue* blocks  = parse(options, tokens);
    return flatten(options, blocks);
}
10 Define the phases
In C you can easily forward declare function, so you don’t have to come up with some clever escabotage like we had to do in F#.

static
GQueue* remove_empty_blocks(Options*, GQueue*);
static
GQueue* merge_blocks(Options*, GQueue*);
static
GQueue* add_code_tags(Options*, GQueue*);

static
GQueue* process_phases(Options* options, GQueue* blocks) {

    blocks          = remove_empty_blocks(options, blocks);
    blocks          = merge_blocks(options, blocks);
    blocks          = add_code_tags(options, blocks);
    return blocks;
}

static
char* extract(Block* b) {
    return  b->kind == Code         ? b->Code.code          :
            b->kind == Narrative    ? b->Narrative.narrative:
                                      g_assert_no_match;
}

There must be a higher level way to write this utility function …

static
bool is_str_all_spaces(const char* str) {
    g_assert(str);
    while(*str != '0') {
        if(!g_ascii_isspace(*str))
            return false;
        str++;
    }
    return true;
}

static
GQueue* remove_empty_blocks(G_GNUC_UNUSED Options* options, GQueue* blocks) {

    g_queue_foreach(blocks, g_func(Block*, b,
        if(is_str_all_spaces(extract(b)))
            g_queue_remove(blocks, b);
                                   ), NULL);
    return blocks;
}

static
GQueue* merge_blocks(G_GNUC_UNUSED Options*options, GQueue* blocks) {
    return  g_queue_is_empty(blocks)            ? blocks            :
            g_queue_get_length(blocks) == 1     ? blocks            :
                ({
                 Block* h1 = g_queue_pop_head(blocks);
                 Block* h2 = g_queue_pop_head(blocks);
                 h1->kind == Code && h2->kind == Code ? ({
                     char* newCode =
                        g_strjoin("", h1->Code.code, NL, h2->Code.code, NULL);
                     Block* b = union_new(Block, Code, .code = newCode);
                     merge_blocks(options, g_queue_push_front(blocks, b));
                                                         })         :
                 h1->kind == Narrative && h2->kind == Narrative ? ({
                     char* newNarr =
                        g_strjoin(
                            "", h1->Narrative.narrative, NL, h2->Narrative.narrative, NULL);
                     Block* b = union_new(Block, Narrative, .narrative = newNarr);
                     merge_blocks(options, g_queue_push_front(blocks, b));
                                                         })         :
                                                         ({
                     GQueue* newBlocks =
                        merge_blocks(options, g_queue_push_front(blocks, h2));
                     g_queue_push_front(newBlocks, h1);
                                                         });
                 });
}

This really should be in glib …

inline static
gint g_asprintf_z(gchar** string, gchar const *format, ...) {
    va_list argp;
    va_start(argp, format);
    gint bytes = g_vasprintf(string, format, argp);
    va_end(argp);
    return bytes;
}

static
char* indent(int n, char* s) {
    g_assert(s);

    char* ind       = g_strnfill(n, ' ');
    char* tmp;
    g_asprintf(&tmp, "%s%s", ind, s);

    char* withNl;
    g_asprintf(&withNl, "n%s", ind);

    return g_strjoinv(withNl, g_strsplit(tmp, NL, -1));
}
And finally I ended up defining map. See if you like how the usage looks in the function below.

#define g_queue_map_z(q, type, name, ...) ({                                
        GQueue* private_res = g_queue_new();                                
        g_queue_foreach(q, g_func(type, name,                               
            name = __VA_ARGS__;                                             
            g_queue_push_tail(private_res, name);                           
            ), NULL);                                                       
        private_res;                                                        
                                      })

static
GQueue* add_code_tags(Options* options, GQueue* blocks) {

    GQueue* indent_blocks(GQueue* blocks) {
        return g_queue_map(blocks, Block*, b,
                b->kind == Narrative ? b                                                                                                    :
                b->kind == Code      ?
                    union_new(Block, Code, .code =
                        indent(options->code_symbols->Indented.indentation, b->Code.code))    :
                    g_assert_no_match;);
    }

    GQueue* surround_blocks(GQueue* blocks) {
        return g_queue_map(blocks, Block*, b,
                b->kind == Narrative ?
                    union_new(Block, Narrative, .narrative =
                        g_strjoin("", NL, g_strstrip(b->Narrative.narrative), NL, NULL))   :
                b->kind == Code      ?
                    union_new(Block, Code, .code = g_strjoin("",
                                                 NL,
                                                 options->code_symbols->Surrounded.start_code,
                                                 NL,
                                                 g_strstrip(b->Code.code),
                                                 NL,
                                                 options->code_symbols->Surrounded.end_code,
                                                 NL,
                                                 NULL))    :
                                       g_assert_no_match;);

    }

    return  options->code_symbols->kind == Indented     ?   indent_blocks(blocks)   :
            options->code_symbols->kind == Surrounded   ?   surround_blocks(blocks) :
                                                            g_assert_no_match;
}

char* stringify(GQueue* blocks) {
    GString* res = g_string_sized_new(2048);
    g_queue_foreach(blocks, g_func(Block*, b,
        g_string_append(res, extract(b));
    ), NULL);
    return g_strchug(res->str);
}

void deb(GQueue* q);

static
char* translate(Options* options, char* source) {
    g_assert(options);
    g_assert(source);

    GQueue* blocks  = blockize(options, source);
    blocks          = process_phases(options, blocks);
    return stringify(blocks);
}

Parsing the command line

In glib there is a command line parser that accept options in unix-like format and automatically produces professional –help messages and such. We shoudl really have something like this in .NET. Pheraps we do and I’m not aware of it?

typedef struct CmdOptions { char* input_file; char* output_file; Options* options;} CmdOptions;

static
CmdOptions* parse_command_line(int argc, char* argv[]);

static char *no = NULL, *nc = NULL, *l = NULL, *co = NULL, *cc = NULL, *ou = NULL;
static char** in_file;

static int ind = 0;
static bool tests = false;

// this is a bug in gcc, fixed in 2.7.0 not to moan about the final NULL
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wmissing-field-initializers"

static GOptionEntry entries[] =
{
  { "language"          , 'l', 0, G_OPTION_ARG_STRING, &l ,
                                "Language used", "L"  },
  { "output"            , 'o', 0, G_OPTION_ARG_FILENAME, &ou,
                                "Defaults to the input file name with mkd extension", "FILE" },
  { "narrative-open"    , 'p', 0, G_OPTION_ARG_STRING, &no,
                                "String opening a narrative comment",   "NO" },
  { "narrative-close"   , 'c', 0, G_OPTION_ARG_STRING, &nc,
                                "String closing a narrative comment",   "NC" },
  { "code-open"         , 'P', 0, G_OPTION_ARG_STRING, &co,
                                "String opening a code block",          "CO" },
  { "code-close"        , 'C', 0, G_OPTION_ARG_STRING, &cc,
                                "String closing a code block",          "CC" },
  { "indent"            , 'i', 0, G_OPTION_ARG_INT,    &ind,
                                "Indent the code by N whitespaces",    "N"  },
  { "run-tests"         , 't', G_OPTION_FLAG_HIDDEN, G_OPTION_ARG_NONE,   &tests,
                                "Run all the testcases", NULL },
  { G_OPTION_REMAINING  ,   0, 0, G_OPTION_ARG_FILENAME_ARRAY, &in_file,
                                "Input file to process",   "FILE" },
  { NULL }
};
#pragma GCC diagnostic pop

Brain damaged way to run tests with a -t hidden option. Not paying the code size price in release.

#ifndef NDEBUG
#include "tests.c"
#endif
Here is my big ass command parsing function. It could use a bit of refactoring …

void destroy_arena_allocator();

static
CmdOptions* parse_command_line(int argc, char* argv[]) {

    GError *error = NULL;
    GOptionContext *context;

    context =
        g_option_context_new ("- translate source code with comemnts to an annotated file");
    g_option_context_add_main_entries (context, entries, NULL);
    g_option_context_set_summary(context, summary(s_lang_params_table));

    if (!g_option_context_parse (context, &argc, &argv, &error))
        report_error("option parsing failed: %s", error->message);

    CmdOptions* opt = g_new(CmdOptions, 1);
    opt->options = g_new(Options, 1);

    #ifndef NDEBUG
    if(tests) {
        int i = run_tests(argc, argv);
        exit(i);
    }
    #endif

    if(!in_file) report_error("No input file");
    opt->input_file = *in_file;

    // Uses input file without extension, adding extension .mkd (assume markdown)
    opt->output_file = ou ? ou :  ({
                                  char* output      = g_strdup(*in_file);
                                  char* extension   = g_strrstr(output, ".");
                                  extension ? ({
                                               *extension = '0';
                                               g_strjoin("", output, ".mkd", NULL);
                                                }) :
                                               g_strjoin("", output, ".mkd", NULL);
                                  });

    if(l) { // user passed a language
        LangSymbols* lang = lang_find_symbols(s_lang_params_table, l);
        if(!lang) report_error("%s is not a supported language", l);

        opt->options->start_narrative  = lang->start;
        opt->options->end_narrative    = lang->end;

    } else {
        if(!no || !nc) report_error("You need to specify either -l, or both -p and -c");

        opt->options->start_narrative  = no;
        opt->options->end_narrative    = nc;
    }

    if(ind) { // user pass    g_option_context_free();
        opt->options->code_symbols = union_new(CodeSymbols, Indented, .indentation = ind);
    } else {
        if(!co || !cc) report_error("You need to specify either -indent, or both -P and -C");
        opt->options->code_symbols =
            union_new(CodeSymbols, Surrounded, .start_code = co, .end_code = cc);
    }

    return opt;
}

Some windows programs (i.e. notepad, VS, …) add a 3 bytes prelude to their utf-8 files, C doesn’t know anything about it, so you need to strip it. On this topic, I suspect the program works on UTF-8 files that contain non-ASCII chars, even if when I wrote it I didn’t know anything about localization.

It should work because I’m just splitting the file when I see a certain ASCII string and in UTF-8 ASCII chars cannot appear anywhere else than in their ASCII position.

char* skip_utf8_bom(char* str) {
    unsigned char* b = (unsigned char*) str;
    return  b[0] == 0xEF && b[1] == 0xBB && b[2] == 0xBF    ? (char*) &b[3]  : // UTF-8
                                                              (char*) b;
}

Not freeing memory (again)

The reason I haven’t been freeing memory all along is because I was planning on using an arena allocator (a kind of linear allocator).

Memory management is fully hortogonal to the style of programming described in this post. You can do it whatever way you prefer, but there is a certain affinity between an arena allocator (or garbage collection) and functional programming because of the temporary objects created in expressions. You could create the temporary objects explicitely, but that would diminish the conciseness of the paradigm.

I have an arena allocator implementation here. In the code below I comment it out so that you don’t have a dependency from that code if you want to try this. The program runs so quickly and it does so little that you can probably let the operating system reclame memory at the end of the process life.

If you ended up integrating this with an editor (i.e. literate programming editing), you’d need to be more careful.

#ifdef ARENA

Arena_T the_arena;

inline static
gpointer arena_malloc(gsize n_bytes) {
    return Arena_alloc(the_arena, n_bytes, __FILE__, __LINE__);
}

inline static
gpointer arena_calloc(gsize n_blocks, gsize n_block_bytes) {
    return Arena_calloc(the_arena, n_blocks, n_block_bytes, __FILE__, __LINE__);
}

inline static
gpointer arena_realloc(gpointer mem, gsize n_bytes) {
    return Arena_realloc(the_arena, mem, n_bytes, __FILE__, __LINE__);
}

void arena_free(G_GNUC_UNUSED gpointer mem) {
    // NOP
}

void set_arena_allocator() {
    GMemVTable vt = (GMemVTable) { .malloc = arena_malloc,      .calloc = arena_calloc,
                                   .realloc = arena_realloc,    .free = arena_free,
                                   .try_malloc = arena_malloc,  .try_realloc = arena_realloc};
    g_mem_set_vtable(&vt);

    the_arena = Arena_new();
}

void destroy_arena_allocator() {
    Arena_dispose(&the_arena);
}

#endif

Summary

I have to say, it didn’t feel too cumbersome to structure C code in a functional way, assuming that you can use GLib and a couple of GCC extensions to the language. It certainly doesn’t have the problems that C++ has in terms of debugging STL failures.

There are a couple of things I don’t like about GLib and I’m working on an hobby project to overcome them. Eventually I’ll post it.

int main(int argc, char* argv[])
{
#ifdef ARENA
    set_arena_allocator();
#endif

    CmdOptions* opt = parse_command_line(argc, argv);

    char* source    = NULL;
    GError* error   = NULL;

    if(!g_file_get_contents(opt->input_file, &source, NULL, &error))
        report_error(error->message);

    source = skip_utf8_bom(source);

    char* text              = translate(opt->options, source);

    if(!g_file_set_contents(opt->output_file, text, -1, &error))
        report_error(error->message);

#ifdef ARENA
    destroy_arena_allocator();
#endif

    return 0;
}

Functional programming in C: Implementation

Fri, 11 Jan 2013 16:37:00 GMT

Cleanup

Let’s start simple with the cleanup function. First we need the usual barrage of includes. G_BEGIN_DECLS allows the header to be linked in C++.

#ifndef L_UTILS_INCLUDED
#define L_UTILS_INCLUDED

#include "glib.h"

G_BEGIN_DECLS

#include <stdlib.h>
#include <stdio.h>
This feature is GCC specific. It uses __attribute((cleanup(f))) where f is the cleanup function. In this case the cleanup function just frees the memory.

#ifdef __GNUC__

 static inline void __autofree(void *p) {
     void **_p = (void**)p;
     free(*_p);
 }

auto_clean is a building block that you can use to plug in your own cleanup function. In the common case of memory allocation, I created a wrapper macro auto_free to make it even easier.

#define auto_clean(f)   __attribute((cleanup(f)))
#define auto_free       auto_clean(__autofree)

Lambdas

I took this one from here.

If you think about it, a lambda is just an expression that returns a function. This macro creates a nested function, called fn, inside a statement expression and returns it. Unfortunately these features are gcc specific.

Remember that lambdas are not allocated on the heap, so you have to be careful on how you used them.

#define lambda(return_type, function_body)                                          
  ({                                                                                
    return_type __fn__ function_body                                                
    __fn__;                                                                         
  })

#endif

Unions

A union type is what you would expect: a struct that contains an unnamed union and a field to specify which type it is. We need the list of types in union_decl to create the kind enum. The usage of VA_ARGS allows to use whatever syntax you want to go into the enum (i.e. specify int values).

Having to specify the the types here is unfortunate as you are going to need to specify it in the union_case macros as well.

I haven’t found another way to do it. If you do, let me know.

#define union_decl(alg, ...)                                                        
typedef struct alg {                                                                
    enum {  __VA_ARGS__ } kind;                                                     
    union {

You specify each type for the union with union_type. That looks pretty good to me.

#define union_type(type, ...)                                                       
    struct type { __VA_ARGS__ } type;

Ideally you shouldn’t need to specify alg here. Perhaps there is a way to avoid doing so.

#define union_end(alg)                                                              
    };} alg;

You can then set the fields on the union type by using the below macro. Notice the usage of the new struct constructor here to allow optional named parameters.

This is a statement, so it cannot go into an expression place. I think I could make it an expression that returns the existing (or a new) union. This is going to be a sceanrio if people are not using gcc statement expressions.

#define union_set(instance, type, ...)                                              
    G_STMT_START {                                                                  
        (instance)->kind     = (type);                                              
        (instance)->type   = (struct type) { __VA_ARGS__ };                         
    } G_STMT_END

This is an utility macro. It is a version of g_assert that you can use in an expression position.

#define g_assert_e(expr) (                                                          
    (G_LIKELY (!expr) ?                                                             
   (void)g_assertion_message_expr (G_LOG_DOMAIN, __FILE__, __LINE__, G_STRFUNC, 
                                             #expr)                             
    : (void) 1) )

And this allows to fill the default case in a match statement implemented as a ternary operator. It prints out a text representation of the expression and returns it.

#define union_fail(...) (g_assert_e(((void)(__VA_ARGS__) , false)), (__VA_ARGS__))

The rest of the code is commented out. It is a macro way to do pattern matching. For me, the ternary operator is simpler, but I left it there in case you want to play with it.

/*
#define union_case_only_s(instance, type, ...)                                      
        G_STMT_START {                                                              
        if((instance)->kind == (type)) {                                            
            G_GNUC_UNUSED struct type* it = &((instance)->type); __VA_ARGS__; }     
        else g_assert_not_reached();                                                
        } G_STMT_END

#define union_case_first_s(alg, instance, type, ...)                                
    G_STMT_START {                                                                  
        alg* private_tmp = (instance);                                              
        if(private_tmp->kind == type) {                                             
            G_GNUC_UNUSED struct type* it = &((private_tmp)->type); __VA_ARGS__; }

#define union_case_s(type, ...)                                                     
        else if(private_tmp->kind == type) {                                        
            G_GNUC_UNUSED struct type* it = &((private_tmp)->type); __VA_ARGS__; }

#define union_case_last_s(type, ...)                                                
        else if(private_tmp->kind == type) {                                        
            G_GNUC_UNUSED struct type* it = &((private_tmp)->type); __VA_ARGS__; }  
            else g_assert_not_reached(); } G_STMT_END

#define union_case_default_s(...)                                                   
        else __VA_ARGS__; } G_STMT_END

// Need to use assert here because g_assert* cannot be used in expressions as it expands to do .. while(0)
#define union_case_only(instance, type, ...)                                        
        ( (instance)->kind == (type) ? (__VA_ARGS__) : (assert(false), __VA_ARGS__) )

#define union_case_first(instance, type, ...)                                       
        ( (instance)->kind == (type) ? (__VA_ARGS__) :

#define union_case(instance, type, ...)                                             
        (instance)->kind == (type) ? (__VA_ARGS__) :

#define union_case_last(instance, type, ...)                                        
        (instance)->kind == (type) ? (__VA_ARGS__) : (assert(false), (__VA_ARGS__)) )
#define union_case_default(...)                                                     
        (__VA_ARGS__) )

*/

G_BEGIN_DECLS

#endif // L_UTILS_INCLUDED

Functional programming in C

Fri, 04 Jan 2013 14:45:00 GMT

This post/program (as I’m writing it in literate style) is a continuation of my previous posts about functional programming in C++. I promise I’m not going to post about doing it in assembly language (I think) ….

I came to like the simplicity of C very much and got interested in how you could write functional code in it.

There is one irritating thing about C as a viable programming language. Microsoft’s compiler support is not good. It just supports ANSI C, not C99 or C11. So, if you want to use more modern idyoms, you got to use gcc or clang. In this post I assume you use gcc. I will point out the gcc specific extensions.

Also, the C standard library is pretty limited, so I decided to use GLib to complement it. I also created some macros to simplify the syntax. I never understood why people like templates and think macros are evil. It takes me all of 5 minutes to do a -E on GCC to debug the result of a macro expansion. With templates, well, that’s different.

So, in summary, this post is about how you can write functional code in C, perhaps with some gcc extensions and certainly with some macro tricks. Let’s call it funkyC (thanks Ian ). I’m going to show how to use it first. Next post I’m going to show how it’s implemented.

Discriminated unions in C

With a bit of macro magic, you can get a decent looking discriminated union syntax. First we need to include the correct headers. lutils.h is where all the macros are defined.

#include <glib.h>
#include <stdio.h>
#include <assert.h>
#include <stdbool.h>
#include <signal.h>
#include <string.h>

#include "lutils.h"

Then you can declare a discriminated union with the syntax below. It suffers from two problems: repeting the list of possible types in union_decl and repeating the name of the discriminated union in union_end. Perhaps there is a way to avoid that, but I haven’t found it.

The syntax for the union_type call is the same as you would use inside a struct declaration. We’ll see how this works when we look at lutils.h.

union_decl  (Car, Volvo, Fiat, Ferrari)
union_type      (Volvo,     int x; double y;)
union_type      (Fiat,      char* brand, *model;)
union_type      (Ferrari,   char* brand, *model;)
union_end   (Car)

We can create a Car either on the stack, as below, or on the heap and we can set its value with union_set.

Notice the usage of the new struct construction syntax to simulate optional named parameters in C. I would prefer not to have a dot at the start of the name, but overall it is beautiful (if I can say that myself).

static void printCar(Car*);

static void testUnion() {
    Car c;

    union_set   (&c, Volvo, .x = 3, .y = 4);
    printCar    (&c);
    union_set   (&c, Ferrari, .brand = "Ferrari");
    printCar    (&c);
    union_set   (&c, Fiat, .brand = "Fiat", .model = "234");
    printCar    (&c);
}

You can then access values inside your discriminated union with normal if statements.

static void testCar(Car*, char const *);

static void printCar(Car* c) {

    if(c->kind == Volvo) {
        int x = c->Volvo.x;
        g_assert_cmpint(x, ==, 3);
    }

Or perhaps you want the equivalent of a match statement in F# (aka an expression that returns a value based on the type of the discriminated union). Notice that, as logical, all the expressions need to return the same type. That’s why union_fail takes a value of the expression type.

    char temp[40];

    char* value =   c->kind == Volvo    ?   itoa(c->Volvo.x, temp, 10)
                  : c->kind == Ferrari  ?   (void)c->Ferrari.model, c->Ferrari.brand
                  : c->kind == Fiat     ?   c->Fiat.model
                                        :   union_fail("Not a valid car type");

If you are willing to be gcc specific, then your expression can be comprised of multiple statements, of which the last one returns the value of the expression. This allows a much more flexible syntax for your match clauses.

#ifdef __GNUC__

    value       =   c->kind == Volvo    ? ({
                                            struct Volvo* it = &c->Volvo;
                                            itoa(it->x, temp, 10);
                                          })
                  : c->kind == Ferrari  ?   (void)c->Ferrari.model, c->Ferrari.brand
                  : c->kind == Fiat     ?   c->Fiat.model
                                        :   union_fail("Not a valid car type");

    testCar(c, value);

#endif // __GNUC__
}

We then use the super simple test framework in GLib to make sure that it all works as expected …

static void testCar(Car* c, char const * value) {
    if(c->kind == Volvo) g_assert_cmpstr(value, ==, "3");
    else if (c->kind == Fiat) g_assert_cmpstr(value, ==, "234");
    else if (c->kind == Ferrari) g_assert_cmpstr(value, ==, "Ferrari");
    else g_assert_not_reached();

}

Nested functions and lambda variables

GCC has many other cool extensions. A very simple one is nested functions. It allows you to nest functions :-) Look at the definition of doA and f2 in the function below. Putting together nested functions and block statement expressions allows you, with some macro magic, to define lambda functions in your code (from here ).

Remember that lambdas (aka nested functions) are allocated on the stack. They are very fast, but you cannot store their pointer into a gloal table (unless such table is used while the stack for this function is alive).

In such cases, you have to create a proper function. But for the other 90% of use cases, they work pretty well. They are lambdas in the spirit of C: very fast, but error prone …

#ifdef __GNUC__

static void testLambda() {

    typedef int (*aFunc) (int);

    aFunc doA(aFunc f){

        int k(int i) {
            return f(i) + 3;
        }
        return k;
    }

    int clos = 2;

    int f2 (int i) {return i;}
    aFunc b = doA(lambda (int, (int p) {return p + clos;}));

    g_assert_cmpint(b(3), ==, 8);
}

Automatic cleanup of local variables

This is not a functional topic per se, but something that always annoyed me tremendously about C. The fact that you cannot define the equivalent of the using statement in C#, or destructors in C++. Well, now you can. Or not?

Again, if you are willing to be GCC specific, you can use an attribute (more on this in the upcoming implementation post) to associate a cleanup function that gets called when your variable goes out of scope. In this case, I wrapped the free case in a nice looking macro.

But that doesn’t really work. You would certainly want such function to be called on any kind of exit from the enclosing scope (aka via exit(), abort() or longjmp()). Alas, that doesn’t happen.

This reduces the usefulness of this mechanism tremendously. Probably too much in that it lulls you into a false sense of security. You still need to free your resources in the error path of your application.

static void testAutomaticCleanup() {
    char* stack_alloc() {
        auto_free char* b = g_malloc(10000);
        memset(b, '#', 10000);
        return b;
    };

    char * c = stack_alloc();
    g_assert(*c != '#');
}

#endif

Data structures

GLib comes with a vast library of data structures to use, not too different from the .NET framework or Java. For example, below you have a single linked list …

static void testGLib() {
     GSList* list = NULL;

     list = g_slist_append(list, "Three");
     list = g_slist_prepend(list, "first");
     g_assert_cmpint(g_slist_length(list), ==, 2);

     list = g_slist_remove(list, "first");
     g_assert_cmpint(g_slist_length(list), ==, 1);

     g_slist_free(list);
}

Wrapping up

There you go, rising the level of abstraction of C, still keeping it very fast (if you are willing to be gcc bound).

There are other features in functional programming languages that are not in this post. Maybe I’ll get around to macro my way into them eventually, maybe not.

In the next post we’ll talk about how all of this is implemented. Below is the code for running the testcases.

int runTests(int argc, char* argv[]) {
    g_test_init(&argc, &argv, NULL);

    if(g_test_quick()) {
        g_test_add_func("/utils/automatic_cleanup", testAutomaticCleanup);
        g_test_add_func("/utils/lambda", testLambda);
        g_test_add_func("/utils/Union", testUnion);
        g_test_add_func("/utils/SList", testGLib);
    }

    return g_test_run();
}

int main(int argc, char *argv[]) {
    return runTests(argc, argv);
}

LLite : language friendly literate programming

Fri, 14 Dec 2012 12:29:26 GMT

Main ideas

My interest in literate programming comes from some realizations on my part:

When I go back to code that I have written some time ago, I don’t remember my reasoning
When I write a blog post, my code seems to be better. Perhaps explaining things to people encourages me to be more precise
I like to think top down, but the compiler forces me to write code bottom up, starting from details and going to higher level concepts

Unhappiness with existing tools

Many of the existing literate programming tools work similarly to the original CWeb.

They have a tangle program that goes over your file and extract something that the compiler can understand
They have a weave program that extracts from your file something that the document generator can understand

This scheme has the unfortunate limitation of breaking your code editor. Given that your file is not a valid code file anymore, the editor starts misbehaving (i.e. intellisense breaks). The debugger starts to get confused (albeit people tried to remediate that with cleaver use of #line. If your language has an interactive console, that would not work either.

A different interpretation

The main idea of this program is to add your narrative to the comment part of a code file by extending the comment tag (i.e. in C you could use /** ). This keeps editor, debugger and interactive console working.

The weave phase as been retained and what you are reading is the program that goes over your code file and extracts a nicely formatted (for this program in ‘markdown’ format) file that can then be translated to HTML, PDF, latex, etc…

You got that? The document you are reading now is the program.

Multi-language, multi-document format

LLite works for any programming language, assuming it has open and close comment character sequences, and any documentation format, assuming it has open and close code character sequences (aka allows you to delimitate your code somehow), or it needs the code to be indented. This document uses markdown (with Pandoc extensions to generate table of contents and titles).

Usage

You invoke the program as documented below. The first set of parameters lets you choose the symbols that delimitate your language comments (or the default symbols below). The second set of parameters lets you choose how your target documentation language treats code. Either it delimits it with some symbols or it indents it.

module LLite

let langParamsTable     = [ "fsharp", ("(*" + "*", "*" + "*)") // The + is not to confuse the parser
                            "c", ("/**", "**/")
                            "csharp", ("/**", "**/")
                            "java", ("/**", "**/")] |> Map.ofList

let languages = langParamsTable |> Map.fold (fun state lang _ -> state + lang + " ") ""

let usage   = sprintf @"
Usage: llite inputFile parameters
where:
One of the following two sets of parameters is mandatory
    -no string : string opening a narrative comment
    -nc string : string closing a narrative comment
or
    -l language: where language is one of (%s)

One of the following two sets of parameters is mandatory
    -co string : string opening a code block
    -cc string : string closing a code block
or
    -indent N  : indent the code by N whitespaces

The following parameters are optional:
    -o outFile : defaults to the input file name with mkd extension" languages


let getLangNoNC lang    =
    match Map.tryFind lang langParamsTable with
    | Some(no, nc) -> no, nc
    | None -> failwith (lang + " is not a valid programming language")

Programming Languages limitations

One of the main tenets of literate programming is that the code should be written in the order that facilitates exposition to a human reader, not in the order that makes the compiler happy. This is very important.

If you have written a blog post or tried to explain a codebase to a new joiner, you must have noticed that you don’t start from the top of the file and go down, but jump here and there trying to better explain the main concepts. Literate programming says that you should write your code the same way. But in our version of it, the compiler needs to be kept happy because the literate file is the code file.

Some ingenuity is required to achieve such goal:

In C and C++ you can forward declare functions and classes, also class members can be in any order
In C#, Java, VB.NET, F# (the object oriented part) you can write class members in any order
In the functional part of F# you do have a problem (see later in this doc)

The F# trick below is used in the rest of the program. You’ll understand its usage naturally by just reading the code

let declare<'a>  = ref Unchecked.defaultof<'a>

Implementation

At the core, this program is a simple translator that takes some code text and return a valid markdown/whatever text. We need to know:

The strings that start and end a narrative comment (input symbols)
How to translate a code block into a document. We support these variations:
- Indented: indent them by N spaces
- Surrounded by startCode/endCode strings

type CodeSymbols =
    | Indent of int                 // indentation level in whitespaces
    | Surrounded of string * string // start code * end code

type Options = {
    startNarrative  : string
    endNarrative    : string
    codeSymbols     : CodeSymbols
}

let translate   = declare<Options -> string -> string>

Going over the parse tree

We need a function that takes a string and returns a list with the various blocks. We can then go over each block, perform some operations and, in the end, transform it back to text

type Block =
| Code      of string
| Narrative of string

let blockize = declare<Options -> string -> Block list>

I could have used regular expressions to parse the program, but it seemed ugly. I could also have used FsParsec, but that brings with it an additional dll. So I decided to roll my own parser. This has several problems:

It is probably very slow
It doesn’t allow narrative comments inside comments, in particular it doesn’t allow the opening comment
It doesn’t allow opening comments in the program code (not even inside a string)

The latter in particular is troublesome. You’ll need to use a trick in the code (i.e. concatenating strings) to foul this program in not seeing an opening comment, but it is inconvenient.

With all of that, it works.

TODO: consider switching to FsParsec

Lexer

The lexer is going to process list of characters. We need functions to check if a list of characters starts with certain chars and to return the remaining list after having removed such chars.

BTW: these functions are polymorphic and tail recursive

let rec startWith startItems listToCheck =
    match startItems, listToCheck with
    | [], _             -> true
    | _ , []            -> false
    | h1::t1, h2::t2  when h1 = h2  -> startWith t1 t2
    | _, _              -> false

let rec remove itemsToRemove listToModify =
    match itemsToRemove, listToModify with
    | [], l             -> l
    | _ , []            -> failwith "Remove not defined on an empty list"
    | h1::t1, h2::t2  when h1 = h2  -> remove t1 t2
    | _, _              -> failwith "itemsToRemove are not in the list"

let isOpening options       = startWith (List.ofSeq options.startNarrative) 
let isClosing options       = startWith (List.ofSeq options.endNarrative)
let remainingOpen options   = remove (List.ofSeq options.startNarrative)
let remainingClose options  = remove (List.ofSeq options.endNarrative)

This is a pretty basic tokenizer. It just analyzes the start of the text and returns what it finds. It also keeps track of the line number for the sake of reporting it in the error message.

let NL = System.Environment.NewLine

type Token =
| OpenComment   of int
| CloseComment  of int
| Text          of string

let tokenize options source =

    let startWithNL = startWith (Seq.toList NL)

    let rec text line acc = function
        | t when isOpening options t    -> line, acc, t 
        | t when isClosing options t    -> line, acc, t
        | c :: t as full                ->
            let line' = if startWithNL full then line + 1 else line
            text line' (acc + c.ToString()) t
        | []                            -> line, acc, [] 
    let rec tokenize' line acc = function
        | []                            -> List.rev acc
        | t when isOpening options t    -> tokenize' line
                                            (OpenComment(line)::acc)  (remainingOpen options t)
        | t when isClosing options t    -> tokenize' line
                                            (CloseComment(line)::acc) (remainingClose options t)
        | t                             ->
            let line, s, t'= text line "" t
            tokenize' line (Text(s) :: acc) t'

    tokenize' 1 [] (List.ofSeq source)

Parser

The parse tree is just a list of Chunks, where a chunk can be a piece of narrative or a piece of code.

type Chunk =
| NarrativeChunk    of Token list
| CodeChunk         of Token list

let parse options source =

    let rec parseNarrative acc = function
        | OpenComment(l)::t         ->
            failwith ("Don't open narrative comments inside narrative comments at line "
                                                                                    + l.ToString())
        | CloseComment(_)::t        -> acc, t
        | Text(s)::t                -> parseNarrative (Text(s)::acc) t
        | []                        -> failwith "You haven't closed your last narrative comment"

    let rec parseCode acc = function
        | OpenComment(_)::t as t'   -> acc, t'
        | CloseComment(l)::t        -> parseCode (CloseComment(l)::acc) t
        | Text(s)::t                -> parseCode (Text(s)::acc) t
        | []                        -> acc, []
    let rec parse' acc = function
        | OpenComment(_)::t         ->
            let narrative, t' = parseNarrative [] t
            parse' (NarrativeChunk(narrative)::acc) t' 
        | Text(s)::t                ->
            let code, t' = parseCode [Text(s)] t
            parse' (CodeChunk(code)::acc) t'
        | CloseComment(l)::t           ->
            failwith ("Don't insert a close narrative comment at the start of your program at line "
                                                                                    + l.ToString())
        | []                -> List.rev acc

    parse' [] (List.ofSeq source)

Flattener

The flattening part of the algorithm is a bit unusual. At this point we have a parse tree that contains tokens, but we want to reduce it to two simple node types containing all the text in string form.

TODO: consider managing nested comments and comments in strings (the latter has to happen in earlier phases)

let flatten options chunks =
    let tokenToStringNarrative = function
    | OpenComment(l) | CloseComment(l)  -> failwith ("Narrative comments cannot be nested at line "
                                                                                    + l.ToString())
    | Text(s)                           -> s

    let tokenToStringCode = function
    | OpenComment(l)                -> failwith ("Open narrative comment cannot be in code at line"
                                                                + l.ToString()) +
                                                 ". Perhaps you have an open comment in" +
                                                 " a code string before this comment tag?"
    | CloseComment(_)               -> string(options.endNarrative |> Seq.toArray)
    | Text(s)                       -> s

    let flattenChunk = function
    | NarrativeChunk(tokens)             ->
        Narrative(tokens |> List.fold (fun state token -> state + tokenToStringNarrative token) "")
    | CodeChunk(tokens)                  ->
        Code(tokens |> List.fold (fun state token -> state + tokenToStringCode token) "")

    chunks |> List.fold (fun state chunk -> flattenChunk chunk :: state) [] |> List.rev

We are getting there, now we have a list of blocks we can operate upon

blockize := fun options source -> source |> tokenize options |> parse options |> flatten options

Narrative comments phases

Each phase is a function that takes the options and a block list and returns a block list that has been ‘processed’ in some way.

type Phase = Options -> Block List -> Block List

let removeEmptyBlocks   = declare<Phase>
let mergeBlocks         = declare<Phase>
let addCodeTags         = declare<Phase>

let processPhases options blockList = 
    blockList
    |> !removeEmptyBlocks   options
    |> !mergeBlocks         options
    |> !addCodeTags         options

We want to manage how many newlines there are between different blocks, because we don’t trust the programmer to have a good view of how many newline to keep from comment blocks and code blocks. We’ll trim all newlines from the start and end of a block, and then add our own.

let newLines = [|'\n';'\r'|]

type System.String with
    member s.TrimNl () = s.Trim(newLines)

Remove empty blocks

There might be empty blocks (i.e. between two consecutive comment blocks) in the file. For the sake of formatting the file beautifully, we want to remove them.

let extract = function
    | Code(text)        -> text
    | Narrative(text)   -> text

removeEmptyBlocks := fun options blocks ->
                        blocks |> List.filter (fun b -> (extract b).TrimNl().Trim() <> "")

Merge blocks

Consecutive blocks of the same kind need to be merged, for the sake of formatting the overall text correctly.

TODO: make tail recursive

let rec mergeBlockList = function
    | []        -> []
    | [a]       -> [a]
    | h1::h2::t -> match h1, h2 with
                   | Code(t1), Code(t2)             -> mergeBlockList (Code(t1 + NL + t2)::t)
                   | Narrative(n1), Narrative(n2)   -> mergeBlockList(Narrative(n1 + NL + n2)::t)
                   | _, _                           -> h1::mergeBlockList(h2::t)

mergeBlocks := fun options blocks -> mergeBlockList blocks

Adding code tags

Each code block needs a tag at the start and one at the end or it needs to be indented by N chars.

let indent n (s:string) =
    let pad = String.replicate n " "
    pad + s.Replace(NL, NL + pad)

addCodeTags := fun options blocks ->
    match options.codeSymbols with
    | Indent(n)         ->
        blocks |> List.map (function Narrative(s) as nar -> nar | Code(s) -> Code(indent n s))
    | Surrounded(s, e)  -> 
        blocks |> List.map (function
                            | Narrative(text)   -> Narrative(NL + text.TrimNl() + NL)
                            | Code(text)        -> Code(NL + s + NL + text.TrimNl() + NL + e + NL))

Flatten again

Once we have the array of blocks, we need to flatten them (transform them in a single string), which is trivial, and then finally implement our original translate function.

let sumBlock s b2 = s + extract b2

let flattenB blocks = (blocks |> List.fold sumBlock "").TrimStart(newLines)

translate := fun options text -> text |> !blockize options |> processPhases options |> flattenB

Parsing command line arguments

Parsing command lines involves writing a function that goes from a sequence of strings to an input file name, output file name and Options record

let parseCommandLine = declare<string array -> string * string * Options>

To implement it, we are going to use a command line parser taken from here. The parseArgs function takes a sequence of argument values and map them into a (name,value) tuple. It scans the tuple sequence and put command name into all subsequent tuples without name and discard the initial (“”,”“) tuple. It then groups tuples by name and converts the tuple sequence into a map of (name,value seq)

For now, I don’t need the ‘value seq’ part as all my parameters take a single argument, but I left it in there in case I will need to pass multiple args later on.

open  System.Text.RegularExpressions

let (|Command|_|) (s:string) =
  let r = new Regex(@"^(?:-{1,2}|\/)(?<command>\w+)[=:]*(?<value>.*)$",RegexOptions.IgnoreCase)
  let m = r.Match(s)
  if m.Success
  then 
    Some(m.Groups.["command"].Value.ToLower(), m.Groups.["value"].Value)
  else
    None


let parseArgs (args:string seq) =
  args 
  |> Seq.map (fun i -> 
                    match i with
                    | Command (n,v) -> (n,v) // command
                    | _ -> ("",i)            // data
                  )
  |> Seq.scan (fun (sn,_) (n,v) -> if n.Length>0 then (n,v) else (sn,v)) ("","")
  |> Seq.skip 1
  |> Seq.groupBy (fun (n,_) -> n)
  |> Seq.map (fun (n,s) -> (n, s |> Seq.map (fun (_,v) -> v) |> Seq.filter (fun i -> i.Length>0)))
  |> Map.ofSeq

let paramRetrieve (m:Map<string,_>) (p:string) = 
  if Map.containsKey p m
  then Some(m.[p])
  else None

This is the main logic of parameter passing. Note that we give precedence to the -l and -indent parameters, if present.

This is a function that goes from the map of command line parameters to the input file name, output file name and options. With that we can finally define the original parseCommandLine.

let safeHead errMsg s = if s |> Seq.isEmpty then failwith errMsg else s |> Seq.head 

let paramsToInputs paramsMap =
    let single p er     = match paramRetrieve paramsMap p with | Some(k) -> Some(k |> safeHead er)
                                                               | None -> None
    let get p s         = match paramRetrieve paramsMap p with |Some(k) -> k |> safeHead s
                                                               | None -> failwith s
    let defaultP p q er = match paramRetrieve paramsMap p with | Some(k) -> k |> safeHead er
                                                               | None -> q

    let inputFile       = get "" "You need to pass an input file"
    let outputFile      = defaultP  "o"
                                    (System.IO.Path.GetFileNameWithoutExtension(inputFile:string) + ".mkd")
                                    "You must pass a parameter to -o"

    let no, nc          = match single "l" "You must pass a language parameter to -l" with
                          | Some(l) -> getLangNoNC l
                          | None    ->
                                get "no" "The no (narrative open) parameter is mandatory, if no -l specified",
                                get "nc" "The nc (narrative close) parameter is mandatory, if no -l specified"

    let codeSymbs       = match single "indent" "You must pass a whitespace indentation number to -indent" with
                          | Some(n)     ->
                                let success, value = System.Int32.TryParse n
                                if success
                                    then Indent(value)
                                    else failwith "-i accepts just an integer value as parameter"                          
                          | None        ->
                                match single "l" "You must pass a language parameter to -l" with
                                | Some(l) -> Surrounded("~~~" + l,"~~~")
                                | None    ->
                                    Surrounded(
                                        get "co" "The co (code open) parameter is mandatory, if no -indent specified",
                                        get "cc" "The cc (code close) parameter is mandatory")
    inputFile, outputFile, {
        startNarrative  = no
        endNarrative    = nc
        codeSymbols     = codeSymbs
        }

parseCommandLine := parseArgs >> paramsToInputs

Main method

We can then write main as the only side effect function in the program. Here is where the IO monad would live …

let banner  = "LLite : language friendly literate programming\n"

let printMemory () =
    let bytesInMeg = 1048576.
    let peak = System.Diagnostics.Process.GetCurrentProcess().PeakWorkingSet64
    let work = System.Diagnostics.Process.GetCurrentProcess().WorkingSet64
    printfn "Peak working set: %A" ((float) peak / bytesInMeg)
    printfn "Working set: %A" ((float) peak / bytesInMeg)

let myMain args =
        try

            let inputFile, outputFile, options = !parseCommandLine args
            let input       = System.IO.File.ReadAllText inputFile
            let output      = !translate options input
            System.IO.File.WriteAllText (outputFile, output)

            //printMemory ()
            0
        with
        | e ->
            printfn "%s" "Failure"
            printfn "%s" e.Message 
            printfn "%s" usage
    #if DEBUG 
            printfn "\nDetailed Error Below:\n%A" e
    #endif
            -1

An aside: forward declaring functions in F#

A simple solution

You can achieve something somehow similar to forward declaration by the ‘declare ‘dirty trick used in this program. Whenever you want to do a forward declaration of a function , or variable, you can type:

let testDeclare() =

    let add = declare<float -> float>

    let ``function where I want to use add without having defined it`` nums = nums |> Seq.map !add

This creates a ref to a function from float to float. It looks a bit like an Haskell type declaration. You can then use such function as if it were actually define and delay his definition to a later point in time when you are ready to explain it.

When you are ready to talk about it, you can then define it with:

    add := fun x -> x + 1.

The syntax is not too bad. You get that often-sought Haskell like explicit type declaration and you can regex the codebase to create an index at the end of the program (maybe).

But is it too slow? After all, there is one more indirection call for each function call.

Let’s test it: enable #time in F# interactive and execute timeNormalF and timeIndirectF varying sleepTime and howManyIter until you convince yourself that it is ok (or not).

    let sleepTime   = 50
    let howManyIter = 100
    let normalF x   = System.Threading.Thread.Sleep sleepTime
    let indirectF   = declare<int -> unit>
    indirectF      := fun x -> System.Threading.Thread.Sleep sleepTime

    let timeNormalF     = [1..howManyIter] |> List.iter normalF
    let timeIndirectF   = [1..howManyIter] |> List.iter !indirectF
    ()

A correct solution (but ugly)

Unfortunately, there is a big problem with all of the above: it doesn’t work with generic functions and curried function invocations. The code below works in all cases, but it is ugly for the user to use. In this program I’ve used the beautiful, but incorrect, syntax.

type Literate() =
    static member Declare<'a, 'b>  (ref : obj ref) (x : 'a) : 'b =
        unbox <| (unbox<obj -> obj> !ref) x
    static member Define<'a, 'b> (func : 'a -> 'b) (ref : obj ref) (f : 'a -> 'b) =
        ref := box (unbox<'a> >> f >> box)

// Declaration    
let rec id (x : 'a) : 'a = Literate.Declare idImpl x
and idImpl = ref null

// Usage
let f () = id 100 + id 200

// Definition
Literate.Define id idImpl (fun x -> x)

The traditional way

The traditional way of doing something like this is by passing the function as a parameter in a manner similar to the below.

// Declaration and usage intermingled
let calculate' (add: int -> int -> int) x y = add x y * add x y

// Definition
let add x y = x + y

let calculate = calculate' add

To my way of seeing, this is the most cumbersome solution and conceptually dishonest because you are not parametrizing your function for technical reasons, but just for the sake of explaining things. In a way, you are changing the signature of your functions for the sake of writing a book. That can’t be right …

[<EntryPoint>]
let main args = myMain args

Exceptions vs. Return Values to represent errors (in F#) – IV – Implementation

Fri, 07 Dec 2012 09:44:00 GMT

The Critical monad is defined as follows. First there is the type that propagates through the monad:

type Result<'a, 'b> =
| Success of 'a
| Failure of 'b

Then we define the usual computation expression methods.

type Critical() =
       // a -> m a
        member o.Return x       = Success x
        // m a -> (a -> m b) -> m b
        member o.Bind (m, f)    = match m with
                                    | Failure e -> Failure e
                                    | Success x -> f x
        // m a -> m a
        member o.ReturnFrom m   = m

Explaining how computational expressions work in F# is a blog onto itself. And several chapters in many books. Sufficient to say that conceptually this propagates the success value and returns the failure value.

We then define an instance of this type, so that we can use the nice ‘critical { … }’ syntax.

let critical = Critical()

We then go and define the functions that the user needs to use to annotate their function calls. The simplest one is the one to propagate any exception coming from the function ‘f’.

let fault f = f

Then it comes the one to manage contingencies. This will trap any exception for which ‘stopF ex’ is ‘true’, call ‘errF ex’ to construct the error return value and wrap it in a ‘Failure’. Otherwise it will rethrow the exception.

let contingentGen stopF errF f =
    try
        Success(f ())
    with
        | ex when stopF ex -> Failure(errF ex)
        | _                -> reraise ()

Albeit very simple, the above is the core of the system. Everything else is just details. Let’s look at them.

First we want a function that takes as parameter a list of (Exception, ReturnValue) and gives back the correct stopF errF to plug into ‘contingentGen’.

let exceptionMapToFuncs exMap =
    let tryFind ex = exMap |> List.tryFind (fun (k, _) -> k.GetType() = ex.GetType())
    (fun ex ->
        let found = tryFind ex
        match found with Some(_) -> true | None -> false),
    (fun ex ->
        let found = tryFind ex
        match found with
        | Some(k, v)    -> v ex
        | None          -> raise ex)

Then ugliness comes. For the sake of getting a decent syntax (not great) on the calling site, we need to fake overloading of functions by the old trick of adding a number at the end. Thanks to Tobias to point out this (my api was even worse earlier).

I often wondered about the trade-off between currying and overloading for functions. I seem to always paint myself in a situation where I need overloading. In any case, here it goes:

let contingent1 exMap f x =
    let stopF, errF = exceptionMapToFuncs exMap
    contingentGen stopF errF (fun _ -> f x)
let contingent2 exMap f x y =
    let stopF, errF = exceptionMapToFuncs exMap
    contingentGen stopF errF (fun _ -> f x y)
let contingent3 exMap f x y z =
    let stopF, errF = exceptionMapToFuncs exMap
    contingentGen stopF errF (fun _ -> f x y z)

Sometimes you want to trap all exceptions from a function and return your own error value:

let neverThrow1 exc f x     = contingentGen (fun _ -> true) (fun ex -> exc ex) (fun _ -> f x)
let neverThrow2 exc f x y   = contingentGen (fun _ -> true) (fun ex -> exc ex) (fun _ -> f x y)
let neverThrow3 exc f x y z = contingentGen (fun _ -> true) (fun ex -> exc ex) (fun _ -> f x y z)

Other times you need to go from a function that returns return values to one that throws exceptions. You need translating from contingencies to faults:

let alwaysThrow exc f x =
    match f x with
    | Success(ret)              -> ret
    | Failure(e)                -> raise (exc e)

And that’s it. Hopefully we have bridged the gap between exceptions and return values without making the code too ugly (just a little bit). Or perhaps not.

I need to add that I haven’t used this library myself (yet). I’m sure when I do I’ll discover many things to change.

Exceptions vs. Return Values to represent errors (in F#) – III–The Critical monad

Fri, 30 Nov 2012 16:41:00 GMT

Code for this post is here.

In the last post we looked at some Critical code and decided that, albeit correct, it is convoluted. The error management path obfuscates the underlying logic. Also we have no way of knowing if a developer had thought about the error path or not when invoking a function.

Let’s tackle the latter concern first as it is easier. We want the developer to declaratively tag each method call with something that represents his intent about managing the Contingencies or Faults of the function. Moreover if the function has contingencies, we want to force the developer to manage them explicitly.

We cannot use attributes for this as function calls happen in the middle of the code, so there is no place to stick attributes into. So we are going to use higher level functions to wrap the function calls.

The first case is easy. If the developer thinks that the caller of his code has no way to recover from all the exceptions thrown by a function, he can prepend his function call with the ‘fault’ word as in:

fault parseUser userText

That signals readers of the code that the developer is willing to propagate up all the exceptions thrown by the function parseUser. Embarrassingly, ‘fault’ is implemented as:

let fault f = f

So it is just a tag. Things get trickier when the function has contingencies. We want to find a way to manage them without introducing undue complexity in the code.

We’d like to catch some exceptions thrown by the function and convert them to return values and then either return such return values or manage the contingency immediately after the function call. On top of that, we’d want all of the code written after the function call to appear as clean as if no error management were taking place. Monads (computation values) can be used to achieve these goals.

Last time we introduced a type to represent error return values:

type Result<'a, 'b> =
| Success of 'a
| Failure of 'b

type UserFetchError =
| UserNotFound  of exn
| NotAuthorized of int * exn

We can then create a computation expression that ‘abstracts out’ the Failure case and let you write the code as cleanly as if you were not handling errors. Let’s call such thing ‘critical’. Here is how the final code looks like:

let tryFetchUser3 userName =
    if String.IsNullOrEmpty userName then invalidArg "userName" "userName cannot be null/empty"
    critical {
        let Unauthorized (ex:exn) = NotAuthorized (ex.Message.Length, ex)
        let! userText = contingent1
                            [FileNotFoundException()        :> exn, UserNotFound;
                             UnauthorizedAccessException()  :> exn, Unauthorized]
                            dbQuery (userName + ".user")
        return fault parseUser userText
    }

You can compare this with the code you would have to write without the ‘critical’ library (from last post):

let tryFetchUser1 userName =
    if String.IsNullOrEmpty userName then invalidArg "userName" "userName cannot be null/empty"
    // Could check for file existence in this case, but often not (i.e. db)
    let userResult =    try
                            Success(dbQuery(userName + ".user"))
                        with
                        | FileNotFoundException as ex        -> Failure(UserNotFound ex)
                        | UnauthorizedAccessException as ex  -> Failure(NotAuthorized(2, ex))
                        | ex                                    -> reraise ()
    match userResult with
    | Success(userText) ->
        let user        = Success(parseUser(userText))
        user
    | Failure(ex)       -> Failure(ex)

And with the original (not critical) function:

let fetchUser userName =
    let userText            = dbQuery (userName + ".user")
    let user                = parseUser(userText)
    user

Let’s go step by step and see how it works. First of all, you need to enclose the Critical parts of your code (perhaps your whole program) in a ‘critical’ computation:

critical {
   ...
}

This allows you to call functions that return a Result and manage the return result as if it were the successful result. If an error were generated, it would be returned instead. We will show how to manage contingencies immediately after the function call later.

The above is illustrated by the following:

let! userText = contingent1
                    [FileNotFoundException()        :> exn, UserNotFound;
                     UnauthorizedAccessException()  :> exn, Unauthorized]
                    dbQuery (userName + ".user")

Here ‘contingent1’ is a function that returns a Result, but userText has type string. The Critical monad, and in particular the usage of ‘let!’ is what allows the magic to happen.

‘contingentN’ is a function that you call when you want to manage certain exceptions thrown by a function as contingencies. The N part represents how many parameters the function takes.

The first parameter to ‘contingent1’ is a list of pairs (Exception, ErrorReturnConstructor). That means: when an exception of type Exception is thrown, return the result of calling ‘ErrorReturnConstructor(Exception)’ wrapped inside a ‘Failure’ object. The second parameter to ‘contingent1’ is the function to invoke and the third is the argument to pass to it.

Conceptually, ‘ContingentN’ is a tag that says: if the function throws one of these exceptions, wrap them in these return values and propagate all the other exceptions. Notice that Unauthorized takes an integer and an exception as parameters while the ErrorReturnConstructor takes just an exception. So we need to add this line of code:

let Unauthorized (ex:exn) = NotAuthorized (ex.Message.Length, ex)

After the contingent1 call, we can then write code as if the function returned a normal string:

return fault parseUser userText

This achieves that we set up to do at the start of the series:

Contingencies are now explicit in the signature of tryFetchUser3
The developer needs to indicate for each function call how he intend to manage contingencies and faults
The code is only slightly more complex than the non-critical one

You can also decide to manage your contingencies immediately after calling a function. Perhaps there is a way to recover from the problem. For example, if the user is not in the database, you might want to add a standard one:

let createAndReturnUser userName = critical { return {Name = userName; Age = 43}}
let tryFetchUser4 userName =
    if String.IsNullOrEmpty userName then invalidArg "userName" "userName cannot be null/empty"
    critical {
        let Unauthorized (ex:exn) = NotAuthorized (ex.Message.Length, ex) // depends on ex
        let userFound = contingent1
                            [FileNotFoundException()        :> exn, UserNotFound;
                             UnauthorizedAccessException()  :> exn, Unauthorized]
                            dbQuery (userName + ".user")
        match userFound with
        | Success(userText)         -> return  fault parseUser userText
        | Failure(UserNotFound(_))  -> return! createAndReturnUser(userName)
        | Failure(x)                -> return! Failure(x)
    }

The only difference in this case is the usage of ‘let’ instead of ‘let!’. This exposes the real return type of the function allowing you to pattern match against it.

Sometimes a simple exception to return value mapping might not be enough and you want more control on which exceptions to catch and how to convert them to return values. In such cases you can use contingentGen:

let tryFetchUser2 userName =
    if String.IsNullOrEmpty userName then invalidArg "userName" "userName cannot be null/empty"
    critical {
        let! userText = contingentGen
                            (fun ex -> ex FileNotFoundException || ex UnauthorizedAccessException)
                            (fun ex ->
                                match ex with
                                       | FileNotFoundException       -> UserNotFound(ex)
                                       | UnauthorizedAccessException -> NotAuthorized(3, ex)
                                       | _ -> raise ex)
                            (fun _ -> dbQuery (userName + ".user"))
        return fault parseUser userText
    }

The first parameter is a lambda describing when to catch an exception. The second lambda translate between exceptions and return values. The third lambda represents which function to call.

Sometimes you might want to catch all the exceptions that a function might throw and convert them to a single return value:

type GenericError = GenericError of exn
// 1. Wrapper that prevents exceptions for escaping the method by wrapping them in a generic critical result
let tryFetchUserNoThrow userName =
    if String.IsNullOrEmpty userName then invalidArg "userName" "userName cannot be null/empty"
    critical {
        let! userText = neverThrow1 GenericError dbQuery (userName + ".user")
        return fault parseUser userText
    }

And sometimes you might want to go the opposite way. Given a function that exposes some contingencies, you want to translate them to faults because you don’t know how to recover from them.

let operateOnExistingUser userName =
    let user = alwaysThrow GenericException tryFetchUserNoThrow userName
    ()

Next time we’ll look at how the Critical computation expression is implemented.

Exceptions vs. Return Values to represent errors (in F#) – II– An example problem

Fri, 23 Nov 2012 10:45:00 GMT

In the previous post, we talked about the difference between Critical and Normal code. In this post we are going to talk about the Critical code part. Ideally, we want:

A way to indicate that a particular piece of code (potentially the whole program) is Critical
A way to force/encourage the programmer to make an explicit decision on the call site of a function on how he wants to manage the error conditions (both contingencies and faults)
A way to force/encourage the programmer to expose contingencies/faults that are appropriate for the conceptual level of the function the code is in (aka don’t expose implementation details for the function, i.e. don’t throw SQLException from a getUser method where the caller is supposed to catch it)

Remember that I can use the word ‘force’ here because the programmer has already taken the decision to analyse each line of code for error conditions. As we discussed in the previous post, In many/most cases, such level of scrutiny is unwarranted.

Let’s use the below scenario to unravel the design:

type User = {Name:string; Age:int}

let fetchUser userName =
    let userText            = dbQuery (userName + ".user")
    let user                = parseUser(userText)
    user

This looks like a very reasonable .NET function and it is indeed reasonable in Normal code, but not in Critical code. Note that the caller likely needs to handle the user-not-in-repository case because there is no way for the caller to check such condition beforehand without incurring the performance cost of two network roundtrips.

Albeit the beauty and simplicity, there are issues with this function in a Critical context:

The function throws implementation related exceptions, breaking encapsulation when the user needs to catch them
It is not clear from the code if the developer thought about error management (do you think he did?)
Preconditions are not checked, what about empty or null strings?

To test our design let’s define a fake dbQuery:

let dbQuery     = function
    | "parseError.user"     -> "parseError"
    | "notFound.user"       -> raise (FileNotFoundException())
    | "notAuthorized.user"  -> raise (UnauthorizedAccessException())
    | "unknown.user"        -> failwith "Unknown error reading the file"
    | _                     -> "FoundUser"

The first two exceptions are contingencies, the caller of fetchUser is supposed to manage them. The unknown.user exception is a fault in the implementation. parseError triggers a problem in the parseUser function.

ParseUser looks like this:

let parseUser   = function
    | "parseError"          -> failwith "Error parsing the user text"
    | u                     -> {Name = u; Age = 43}

Let’s now create a test function to test the different versions of fetchUser that we are going to create:

let test fetchUser =
    let p x                 = try printfn "%A" (fetchUser x) with ex -> printfn "%A %s" (ex.GetType()) ex.Message
    p "found"
    p "notFound"
    p "notAuthorized"
    p "parseError"
    p "unknown"

Running the function exposes the problems described above. From the point of view of the caller, there is no way to know what to expect by just inspecting the signature of the function. There is no differentiation between contingencies and faults. The only way to achieve that is to catch some implementation-specific exceptions.

How would we translate this to Critical code?

First, we would define a type to represent the result of a function:

type Result<'a, 'b> =
| Success of 'a
| Failure of 'b

This is called the Either type, but the names have been customized to represent this scenario. We then need to define which kind of contingencies our function could return.

type UserFetchError =
| UserNotFound  of exn
| NotAuthorized of int * exn

So we assume that the caller can manage the fact that the user is not found or not authorized. This type contains an Exception member. This is useful in cases where the caller doesn’t want to manage a contingency, but wants to treat it like a fault (for example when some Normal code is calling some Critical code).

In such cases, we don’t lose important debugging information. But we still don’t break encapsulation because the caller is not supposed to ‘catch’ a fault.

Notice that NotAuthorized contains an int member. This is to show that contingencies can carry some more information than just their type. For example, a caller could match on both the type and the additional data.

With that in place, let’s see how the previous function looks like:

let tryFetchUser1 userName =
    if String.IsNullOrEmpty userName then invalidArg "userName" "userName cannot be null/empty"
    // Could check for file existence in this case, but often not (i.e. db)
    let userResult =    try
                            Success(dbQuery(userName + ".user"))
                        with
                        | FileNotFoundException as ex        -> Failure(UserNotFound ex)
                        | UnauthorizedAccessException as ex  -> Failure(NotAuthorized(2, ex))
                        | ex                                    -> reraise ()
    match userResult with
    | Success(userText) ->
        let user        = Success(parseUser(userText))
        user
    | Failure(ex)       -> Failure(ex)

Here is what changed:

Changed name to tryXXX to convey the fact that the method has contingencies
Added precondition test, which generates a fault
The signature of the function now conveys the contingencies that the user is supposed to know about

But still, there are problems:

The code became very long and convoluted obfuscating the success code path
Still, has the developer thought about the error conditions in parseUser and decided to let exceptions get out, or did she forget about it?

The return value crowd at this point is going to shout: “Get over it!! Your code doesn’t need to be elegant, it needs to be correct!”. But I disagree, obfuscating the success code path is a problem because it becomes harder to figure out if your business logic is correct. It is harder to know if you solved the problem you set out to solve in the first place.

In the next post we’ll see what we can do about keeping beauty and being correct.

Exceptions vs. Return Values to represent errors (in F#) – Conceptual view

Mon, 19 Nov 2012 13:02:10 GMT

Recently I’ve been reading numerous articles on the age old question of exceptions vs. return values. There is a vast literature on the topic with very passionate opinions on one side or the other. Below is my view on it.

First of all, I’ll define my terms.

Success code path: chunk of code that is responsible to perform the main task of a function, without any regard for error conditions
Contingency: an event that happens during the success code path execution that is of interest for the caller of the function.
- It happens rarely
- It can be described in terms that the caller understands, it doesn’t refer to the implementation of the function
- The caller stands a chance to recover from it
Fault: an event that happens during the success code path execution that is not expected by the caller of the function.
- It happens rarely
- It cannot be described in terms that the caller understands, it requires reference to the implementation of the function
- The caller has no way to recover from it

Examples of the above for a FetchUser(userName) function:

Success code path: the code that retrieves the user from the database
Contingency: the fact that the requested user is not in the database
Fault: userName = null, divide by zero, stack overflow, …

The difference between Contingency and Fault is not sharp in practice and requires common sense, but it is useful none less. When in doubt, it would appear prudent to consider an event as a Contingency, so that the caller gets a chance to recover.

Ideally, you would like a Contingency to be part of the signature of a function, so that the caller knows about it. On the other end, a Fault shouldn’t be part of the signature of a function for two reasons:

The user cannot recover from it
Being part of the signature would break encapsulation by exposing implementation details of the function

The above seems to suggest that Contingencies should be represented as return values and Faults as exceptions. As an aside, in Java the former is represented as checked exceptions, which is part of the signature. We’ll tackle checked exceptions later on.

An important point that is often neglected in the discussions on this topic is that there are two categories of applications: applications that care about Contingencies (Critical apps) and applications that don’t (Normal apps). I am of the opinion that the latter category is the largest.

In many cases you can indeed write just the success code path and, if anything goes wrong, you just clean up after yourself and exit. That is a perfectly reasonable thing to do for very many applications. You are trading off speed of development with stability. Your application can be anywhere on that continuum.

Examples of Normal apps are: build scripts, utility applications, departmental applications where you can fix things quickly on the user machine, intranet web sites, internet web sites that are purely informative, etc …

Examples of Critical apps are: servers, databases, operating systems, web site that sell stuff, etc …

For Normal apps, treating Contingencies as Fault is the right thing to do. You just slap a try … catch around your event loop/ thread/ process and you do your best to get the developer to fix the problem quickly. I think a lot of the angst of the ‘return value crowd’ is predicated on not having this distinction in mind. They are making very valid point regarding Critical apps to a crowd that is thinking about Normal apps. So the two sides are cross-talking.

Also, in my opinion, the main problem with Java checked exceptions is that they make writing Normal apps as cumbersome as writing Critical apps. So, reasonably, people complain.

The .NET framework decided to use Exceptions as the main way to convey both Faults and Contingencies. By doing so, it makes it easier to write Normal apps, but more difficult to write Critical apps.

For a Critical app or section of code, you’d want to:

Express contingencies in the function signature
Force/Encourage people to take an explicit decision at the calling site on how they want to manage both Contingencies and Faults for each function call

In the next post, let’s see how we can represent some of this in F#.

Retrieving SQL Server data with type providers and exposing it with ASP.NET Web APIs in F#

Tue, 04 Sep 2012 10:16:17 GMT

For a good introduction on how to use Web APIs in F#, read here. The starting point for type providers is here. This post is about how I solved a practical problem using these technologies.

First, let’s discuss the scenario. In my company, we needed to log usage information for our various applications to a central repository and build a web site to access such information. I went through three different architectures for such a requirement, ending with the set of technologies described in the title.

I found the intersection of the SQL type provider with the ASP.NET Web Api to be very sweet. Personally, I think this mix is much better than using the WFC Data Services, because the code you have to write and the infrastructure to maintain is significantly less.

I suspect that the F# team and ASP.Net team didn’t talk to each other. It all happens because of a well defined interface (IQueryable) that both teams happen to work against.

1st version, heavy Javascript

SQL Server Express backend
WFC Data Services middle tier in C# (autogenerated REST service from a table, you can query from the JS code)
Plenty of Javascript in the browser, calling back to the REST web services through Ajax

Here is an example of the kind of manipulation done in the JS layer:

function extractVersions(sessions) {
    var versions = {}
    $.each(sessions, function (i, s) {
        if (typeof versions[s.Version] == 'undefined') {
            versions[s.Version] = 0
        } else {
            versions[s.Version] = versions[s.Version] + 1
        }
    })

    var vs = []
    for (var prop in versions) {
        if (versions.hasOwnProperty(prop))
            vs.push({ Version: prop, Sessions: versions[prop] })
    }
    return vs;
}

Pros: 1. Back-end autogenerated 1. I’ve learned JS pretty well

Cons: 1. Plenty of autogenerated code (crap) to debug into when things go wrong 1. Difficult to customize the back-end 1. Somehow slow, even if today’s JS engines are pretty good 1. More time consuming to create and maintain JS code (compared to F# or C#) 1. Aesthetically unsatisfying, as you really would like to do most manipulations on the server

2nd version, SOAP like web services

SQL Server Express backend
SOAP like web services middle tier in C# (coded by hand), still using the WCF Data Services infrastructure
Little Javascript in the browser exclusively for presentation, calling back to the REST web services through Ajax

Here is how one of the web services would look like:

        [WebGet]
        public IEnumerable<VersionSessions> GetVersionsForMonth(string application, 
                                                                bool isQS, bool isAD)
        {
            var sessions = GetSessionsForMonth(application, isQS, isAD);
            var hash = new Dictionary<string, VersionSessions>();

            foreach (var s in sessions)
            {
                VersionSessions vs;
                var found = hash.TryGetValue(s.Version, out vs);

                if (!found)
                    hash[s.Version] = new VersionSessions { Version = s.Version, 
                                                                      Sessions = 0 };
                else
                    vs.Sessions += 1;
            }
            return hash.Values;
        }

Pros:

Maintainable business logic code (C# instead of JS)
Somehow faster
Easier to customize the back-end (just add another web service with the signature you want)

Cons:

Still plenty of manipulation code (at least now on the back-end), not very RESTy
Feels that the webservice are very ad-hoc for this particular presentation (i.e. a ViewModel, not a model)
Still not trivial to customize the back-end logic, mostly because I was using WCF Data Services, which are opaque

3rd version, using SQL Views and F# type providers

Moved the whole app to a dedicated VM
SQL Server backend with a significant layer of Views and inline table-value functions
Proper REST web services created through ASP.NET Web API and F# type providers
Little Javascript in the browser exclusively for presentation, calling back to the REST web services through Ajax
Moved all external ‘reference file’ (i.e. people in QS) to the DB. Before I was keeping business info in the database and config info in files.

Here is an example of an inline table value function:

ALTER FUNCTION [dbo].[FUsers] (@App nvarchar(40))
RETURNS TABLE
AS
RETURN
(
SELECT     UserAlias, MAX(Application) AS Application, MAX(Start) AS LastSessionStart, MAX(Finish) AS LastSessionFinish, DATEDIFF(MINUTE, MAX(Start), MAX(Finish)) AS LastSessionTotalTime, COUNT(*) AS Sessions, MAX(Machine) AS Machine, MAX(Version) AS Version, MAX(Qs) AS Qs
FROM         dbo.VTotalSessions AS s
WHERE     (TotalTime > 30) and (Application = @App)
GROUP BY UserAlias)

And here is an example of a type provider based web service. Most of my code works like this:

type usersController() =
    inherit ApiController()
 
    let db = dbSchema.GetDataContext()

    member x.Get() =
        query {
            for u in db.VUsers do
            select u
        }

    member x.Get(app) =
        query {
            for u in db.FUsers(app) do
            select u
        }

    member x.Post (value:string) = ()
    member x.Put (id:int) (value:string) = ()
    member x.Delete (id:int) = ()

Here is how a query is represented in JS (the whole query gets passed through to SQL, both the JS part and the F# part, IQueryable magick):

function loadOpenSessions(app, qs,  cont) {

    var query = new oQuery()
        .From("/users/" + app)
        .Let("lastMonth", lastMonth)
        .Let("twoMinutesAgo", twoMinutesAgo)
        .Where("item => item.LastSessionFinish > $twoMinutesAgo")
        .Orderby("UserAlias")
    commonCall(query, qs, cont)
}

Pros:

Purely functional business logic code (SQL), very easy to debug problems by just running the query/view
Maximum usage of SQL Server optimizer. As they say in the SQL Engine Team: “we’ve spent decades optimizing sql query engines. You are unlikely to do better with your ‘for’ loops …”
Very easy to customize the back-end, just write F# code to implement GET/POST/PUT/DELETE etc…
Moving all the state for the app (reference files included) in the DB, makes it much easier to integrate it into the business logic. It all becomes a big query for the optimizer to sort out.
No autogenerated code anywhere in the architecture
More separation between model and viewmodel. Tables are model, Views are ViewModel, F# is just a way to expose such view model to the world at large

Cons:

Routing algorithm for ASP.NET Web API is mysterious. It’s the only black box piece in the architecture (aka that I don’t understand it).
Sometimes either SQL is not powerful enough (I abhor Stored procedures on religious grounds) or something doesn’t map well to a REST metaphor. In such cases I have diluted the metaphor as in the code below. The good thing is that, being just code, I can do that. I’m not blocked.

type statsController() =

    inherit ApiController()

    let db = dbSchema.GetDataContext()

    member x.Get(app:string, isQs:string, call:string) =
        match call with
        | "WeeklyUsers" ->
            let sessions = query {
                for s in db.VTotalSessionsAndWeeks do
                where (s.Application = app && s.Qs = Nullable (Int32.Parse(isQs)))
                select s
            }
            weeklyUsers sessions
        | "WeeklyTime"  ->
            let sessions = query {
                for s in db.FTotalTime(app, Nullable (Int32.Parse(isQs))) do
                select s
            }
            weeklyTime sessions
        | "HourlyUsers"->
            let users = query {
                for u in db.FHourlyUsers(app, Nullable (Int32.Parse(isQs))) do
                select u }
            hourlyUsers users

        | _ -> failwith "No call with that name"

And now Andy has a blog …

Mon, 18 Jun 2012 07:26:49 GMT

For those of you interested in C++, Andy Sawyer now has a blog here. Here is an extended discussion on how to choose an STL container. Enjoy !!

Writing functional code in C++ V – Miscellaneous and conclusions

Fri, 01 Jun 2012 06:45:26 GMT

Just a couple of trivialities and my parting thoughts.

Nested functions

If your language has lambdas, you don’t need nested functions support because you can implement them using it.

I am a heavy user of nested functions, but I’m of two minds about it. On one side, I like that they sit close to where they are used, avoiding going outside the main function body to understand them. I also like that you don’t need to pass a lot of parameters to them, as they capture the function locals. On the other side, they end up creating the impression that your functions are very long and so, in my eyes, they occasionally reduce readability. The IDE helps you out there (at least VS 11) by allowing you to collapse the lambdas.

An example of trivial case is below:

BOOST_AUTO_TEST_CASE(NestedFunctions)
{
    int x = 3;
    auto sumX = [&] (int y) { return x + y;};
    BOOST_CHECK_EQUAL(sumX(2), 3+2);
}

And here is a more realistic one (not written in a functional style), where readability is reduced by the three nested functions (among many other things):

bool condor(
            boost::gregorian::date now,  // date to evaluate
            double spot,                 // spot price underlying
            double v,                    // ATM volatility
            double r,                    // risk free rate
            double step,                 // % of spot price to keep as distance between wings
            int minCallShortDist,        // min distance from the short call strike in steps
            int minPutShortDist,         // min distance from the short put strike in steps
            int minDays,                 // min number of days to expiry
            int maxDays,                 // max number of days to expiry
            double maxDelta,             // max acceptable delta value for shorts in steps
            double minPremium,           // min accepted premium as % of step
            Condor& ret                  // return value
            )
{
    // convert params to dollar signs
    auto stepPr            = round(step * spot);
    auto toUSD             = [stepPr] (double x) { return round(stepPr * x);};
    auto minCpr            = toUSD( minCallShortDist );
    auto minPpr            = toUSD( minPutShortDist );
    auto premiumPr         = toUSD( minPremium );
    // calc strike values for short legs
    auto atm               = round(spot / stepPr) * (long) stepPr;
    auto callShort         = atm + minCpr;
    auto putShort          = atm - minPpr;
    auto addDays           = [](boost::gregorian::date d, int dys) -> boost::gregorian::date {
        using namespace boost::gregorian;
        auto toAdd         = days(dys);
        auto dTarget       = d + toAdd;
        return  dTarget;
    };
    // calc min & max allowed expiry dates
    auto minDate           = addDays(now, minDays);
    auto maxDate           = addDays(now, maxDays);
    auto expiry            = calcExpiry(now, 0);
    // find first good expiry
    while(expiry < minDate)
        expiry          = calcExpiry(expiry, +1);
    Greeks g;
    auto scholes           = [=, &g] (double strike, int days, bool CorP) {
        return blackScholesEuro(spot, strike, days, CorP, v, r, g, true);
    };
    // find a condor that works at this expiry
    auto findCondor        = [=, &g, &ret] (int days) -> bool {
        ret.shortCallStrike                = callShort;
        ret.shortPutStrike                 = putShort;
        auto shCallPremium                 = 0.0;
        auto shPutPremium                  = 0.0;
        // find short call strike price < maxDelta
        while(true) {
            shCallPremium                  = scholes(ret.shortCallStrike, days, true);
            if(g.delta <= maxDelta)
                break;
            else
                ret.shortCallStrike        += stepPr;
        }
        // find short put strike price < maxDelta
        while(true) {
            shPutPremium                   = scholes(ret.shortPutStrike, days, false);
            if( (- g.delta) <= maxDelta)
                break;
            else
                ret.shortPutStrike         -= stepPr;
        }
        // check premium is adeguate
        ret.longCallStrike                = ret.shortCallStrike + stepPr;
        ret.longPutStrike                 = ret.shortPutStrike  - stepPr;
        auto lgCall                       = scholes(ret.longCallStrike, days, true);
        auto lgPut                        = scholes(ret.longPutStrike,  days, false);
        ret.netPremium                    = shCallPremium + shPutPremium - lgCall - lgPut;
        return ret.netPremium > premiumPr;
    };
    // increases the expiry until it finds a condor or the expiry is too far out
    while (expiry < maxDate) {
        auto days        = (expiry - now).days();
        if(findCondor(days)) {
            ret.year     = expiry.year();
            ret.month    = expiry.month();
            ret.day      = expiry.day();
            return true;
        }
        expiry           = calcExpiry(expiry, +1);
    }
    return false;
}

That is quite a mouthful, isn’t it? But really the function is not that long. It is all these nested functions that makes it seems so.

Tuples

Tuples are another feature toward which I have mixed feelings. They are really useful to return multiple results from a function and for rapid prototyping of concepts. But they are easy to abuse. Creating Records is almost always a better, safer way to craft your code, albeit more verbose.

The standard C++ has an excellent tuple library that makes working with them almost as easy as in mainstream functional languages. The below function shows creating them, getting their value, passing them to functions and deconstructing them:

BOOST_AUTO_TEST_CASE(Tuples)
{
    auto t = make_tuple("bob", "john", 3, 2.3);
    BOOST_CHECK_EQUAL(get<0>(t), "bob");
    BOOST_CHECK_EQUAL(get<2>(t), 3);
    // yep, compiler error
    //auto k = get<10>(t);
    auto t2(t);
    BOOST_CHECK(t2 == t);
    // passing as argument, returning it
    auto f = [] (tuple<string, string, int, double> t) { return t;};
    auto t1 = f(t);
    BOOST_CHECK(t == t1);
    // automatic deconstruction
    string s1; string s2; int i; double d;
    std::tie(s1, s2, i, d) = f(t);
    BOOST_CHECK_EQUAL(s1, "bob");
    // partial reconstruction
    string s11;
    std::tie(s11, ignore, ignore, ignore) = f(t);
    BOOST_CHECK_EQUAL(s11, "bob");
}

Conclusion

I’m sure there are some fundamental functional features that I haven’t touched on (i.e. currying, more powerful function composition, etc…). Despite that, I think we have enough material here to start drawing some conclusions. To start with, where is C++ deficient compared to mainstream functional languages for the sake of writing functional code ?

Compiler errors: if you make a well natured error, you often get back from the compiler a long page of rubbish if you are using templates (and you are). You put your goggles on and start swimming through it to find the nugget of gold that is the root cause of your problem.
I can hear you C++ gurus screaming: come on, come on, it’s not that bad. After a while you get used to it. You become proficient in rubbish swimming. That makes me think of the story of that guy who lived in a strange neighbourhood where, as soon as you open the door of you house, someone kicks you in the nuts. His wife suggested that they moved to a better part of town, but the man replied: “no worries, I got used to it”.
Syntax oddity: we did our best to make the gods of syntax happy in this series of articles, but we are still far removed from Haskell, F#, etc beauty…
Advanced features: if you go beyond the basics of functional languages, there is a plethora of interesting features that just make your life so much easier and bring your code to an higher level of abstraction. Things like monads (i.e. f# async workflow), type providers, Haskell’s lazy execution, Haskell’s super strong type system, Closure’s transactional memory etc… Boost is trying to bring many of these things to C++, but there is still a big gap to be filled.
Temptation to cheat: you can cheat in many functional programming languages (i.e. having mutable variables), but nowhere the temptation is as strong as in C++. You need an iron discipline not to do it. The fact that sometimes cheating is the right thing to do makes it even worse.

Overall, am I going to use C++ now as my main programming language? Not really. I don’t have one of those. My general view is to always use the best language for the job at hand (yes, I even use Javascript for web pages).

C++ has some unique characteristics (i.e. speed, cross compilation & simple C interfacing). I’m going to use it whenever I need these characteristics, but now I’ll be a happier user knowing that I can write decent functional code in it.

Writing functional code in C++ IV – Algebraic datatypes

Fri, 27 Apr 2012 09:29:00 GMT

And here comes the guilt bit. I have the strong suspicion (but not certainty) that what I am doing here can be done with templates, but didn’t take the time to do it. With that out of the way, let’s go.

Code for this post is here. Thanks to Steve Bower and Andy Sawyer for reviewing it.

Algebraic datatypes (discriminated unions in F#) are a powerful concept in functional programming. They are the main way to represent type variation in your program. Very roughly, where object orientation uses derivation, functional programming uses algebraic datatypes. An entire book could be written on the theory of this, but the goal of this post is to see how we can map them to C++ without loosing C++ness.

When talking about this with C++ programmers, they always point me to boost variant. That doesn’t do it for me for several reasons.

First of all, boost variants represent one of a fixed collection of types. Algebraic datatypes represent one of a fixed collection of named types. That means that a simple thing like the code below cannot be represented as variant:

type LivingEntity =
| Person of string  // name
| Dog of string     // name

Ok, ok maybe you could represent it by ‘typifing’ things using boost strong typedef, but things get ugly syntactically. Moreover, a lot of time the name is all you care about …

type Switch = On | Off

Are we going to strong typedef for such a simple thing? Oh boy. Even accepting the syntactic weight of all this, there are other issues in play. Discriminated unions are used extensively in functional programs. So you want a nice syntax to work with them Something like the below F# syntax:

let print living =
    match living with
    | Person(name) -> printfn "I'm a per named %s" name
    | Dog(name)    -> printfn "I'm a dog named %s" name

Which could be made even sweeter by using the ‘function’ keyword as below:

let print = function
    | Person(name) -> printfn "I'm a per named %s" name
    | Dog(name)    -> printfn "I'm a dog named %s" name

In boost variant, you either use the get functions or you write a visitor function. In the first case you are probably going to write a chain of ‘if’ statements or a ‘switch’ statement. Both are confusing and come with plenty of syntactic weight. I don’t really want to write a visitor like the one below for each ‘match’ in my code. The magic is gone.

class times_two_visitor
    : public boost::static_visitor<>
{
public:
    void operator()(int & i) const
    {
        i *= 2;
    }
    void operator()(std::string & str) const
    {
        str += str;
    }
};

Ok, so boost variant doesn’t really work for this. Remember that our overarching goal was to stay close to C++. The language itself has something that comes pretty close to what we want in the form of a union, or better a tagged union. Again, the types are not named, but maybe we can work that in.

It turns out that Jared here did all the hard work. The general idea is to use macros to hide the construction of a tagged union with methods to test the type and return the contained value.

For example this code:

DU_DECLARE(LivingEntity)
    DU_VALUE(Person,    string)
    DU_VALUE(Dog,       string)
DU_END

Becomes something like:

struct LivingEntity {
    private:
        LivingEntity() {}
        unsigned int m_kind;
    public:
        static LivingEntity Person(const string& value) {
            LivingEntity unionValue;
            unionValue.m_kind = 19;
            unionValue.m_Person = value;
            return unionValue; }
        bool IsPerson() const {
            return m_kind == 19;
        }
        const string& GetPerson() const {
            (void)( (!!(m_kind == 19)) || (_wassert(L"m_kind == __LINE__", L"c:discriminated_union.cpp", 19), 0) );
            return m_Person; }
        string GetPerson() {
            (void)( (!!(m_kind == 19)) || (_wassert(L"m_kind == __LINE__", L"c:discriminated_union.cpp", 19), 0) );
            return m_Person; }
   private:
        string m_Person;

   …

You can see the outline of a tagged union (i.e. m_kind) with a constructor for each type (i.e. Person) and methods to test for at type and return its value. You can also see the storage for the value (i.e. m_Person).

The only thing in DU_DECLARE that is different from Jared’s solution is the typedef below that allows not repeating the LivingEntity name in each DU_VALUE.

#define DU_DECLARE(name)                        \
    struct name {                               \
    private:                                    \
        typedef name unionName;                 \
        name() {}                               \
        unsigned int m_kind;                    \
    public:

#define DU_VALUE(entryName, entryType)                                                                      \
        static unionName entryName(const entryType& value) {                                                \
            unionName unionValue;                                                                           \
            unionValue.m_kind = __LINE__;                                                                   \
            unionValue.m_##entryName = value;                                                               \
            return unionValue;  }                                                                           \
        bool Is##entryName() const { return m_kind == __LINE__;}                                            \
        const entryType& Get##entryName() const { assert(m_kind == __LINE__); return m_##entryName; }       \
        entryType Get##entryName() { assert(m_kind == __LINE__); return m_##entryName; }                    \
    private:                                                                                                \
        entryType m_##entryName;                                                                            \
    public:

With all of that at your disposal it becomes easy to write:

auto entity = LivingEntity::Dog("Bob");
DU_MATCH(entity)
    DU_CASE(Dog,   BOOST_CHECK_EQUAL(value, "Bob");)
    DU_CASE(Person,BOOST_CHECK(false);)
DU_MATCH_END

There are some beautiful things about this. First of all, the construction of any of such types is super simple. You even get intellisense!

Moreover the ‘value’ variable contains whatever was passed in the constructor for the object. So this is semantically equivalent, if not syntactically, to the match statement in F#.

Obviously the code part is not limited to a single instruction:

DU_MATCH(entity)
    DU_CASE(Dog,
        cout << "I should be here";
        BOOST_CHECK_EQUAL(value, "Bob");
    )
    DU_CASE(Person,
        BOOST_CHECK(false);
    )
DU_MATCH_END

And for those of you addicted to braces, I venture:

DU_MATCH(entity)
    DU_CASE(Dog,
    {
        cout << "I should be here";
        BOOST_CHECK_EQUAL(value, "Bob");
    })
    DU_CASE(Person,
    {
        BOOST_CHECK(false);
    })
DU_MATCH_END

They all work with the same macro definition. They expand to something along the line of:

if(false) {}
    else if(entity.IsDog()) {
        auto value = entity.GetDog();
        BOOST_CHECK_EQUAL(value, "Bob");
    }
    else if(entity.IsPerson()) {
        auto value = entity.GetPerson();
        BOOST_CHECK(false);
    }
    else {
        throw match_exception();
    }

I’ve not reached the pinnacle of macro naming mastering with this one. Making them lowercase and risking a bit more on the conflict side would make the syntax much more palatable. I call it, as it is, not too bad.

The last ‘else’ clause assures you then if you add a new type to the discriminated union and forget to update one of the ‘MATCH’ clauses at least you get a run time error. That is not good. Functional languages would give you a compile time error, which is much better. Maybe with judicious use of templates you can bring the error at compile time.

The macros are trivial:

class match_exception: std::exception {};
#define DU_MATCH(unionName) { auto du_match_var = unionName; if(false) {}
#define DU_CASE_TAG(entry, ...)                        \
    else if(du_match_var.Is##entry()) {                \
        __VA_ARGS__                                    \
    }
#define DU_CASE(entry, ...)                            \
    else if(du_match_var.Is##entry()) {                \
        auto value = du_match_var.Get##entry();        \
        __VA_ARGS__                                    \
    }
#define DU_DEFAULT(...)                                \
    else if(true) { __VA_ARGS__}
#define DU_MATCH_END else {throw new match_exception();} }

Let’s now go back to our initial goal and see how far off we are. We were trying to do the following:

type LivingEntity =
| Person of string
| Dog of string
let print = function
    | Person(name) -> printfn "I'm a per named %s" name
    | Dog(name)    -> printfn "I'm a dog named %s" name

And here is what we ended up with:

DU_DECLARE(LivingEntity)
    DU_VALUE(Person,    string)
    DU_VALUE(Dog,        string)
DU_END
auto print(LivingEntity en) -> void {
    DU_MATCH(entity)
        DU_CASE(Dog,    cout << "I'm a dog named " << value;)
        DU_CASE(Person, cout << "I'm a per named " << value;)
    DU_MATCH_END
}

In our Switch case:

type Switch = On | Off

You get the good looking:

DU_DECLARE(Switch)
    DU_FLAG(On)
    DU_FLAG(Off)
DU_END

And along the way we lost compile time type safety in the very common case of adding new types to the discriminated union. That’s bad.

Also some of you would strongly dislike the (ab)use of macros. As for me, it looks workable.

Writing functional code in C++ III – Performance of different allocation schemes

Mon, 16 Apr 2012 09:11:00 GMT

In this post we are going to look at the performance characteristics of different ways to allocate objects in C++. The goal is to understand what is the best way to allocate objects in a functional style. Code for this post is here.

First of all, let’s define what we are going to allocate:

struct Record {
    int Id;
    int Value1;
    int Value2;
    int Value3;
    int Value4;
    int Value5;
    int Value6;
    int Value7;
    int Value8;
    int Value9;
};

And a function to initialize it:

void record_init(Record& r, int i) {
    r.Id = i;
    r.Value1 = i;
    r.Value2 = i;
    r.Value3 = i;
    r.Value4 = i;
    r.Value5 = i;
    r.Value6 = i;
    r.Value7 = i;
    r.Value8 = i;
    r.Value9 = i;
}

And a function to convert it to an int (so that the compiler doesn’t optimize away our code):

struct to_int {
    typedef int result_type;
    int operator() (const Record& r) const { return r.Id;}
};

Let’s also define some constants:

const int repetitions = 1000;
const int bigBlock = 1000;

And a timer class to measure performance:

typedef boost::chrono::process_cpu_clock the_clock;
struct timer {
    timer() : start(the_clock::now()) {}
    ~timer() {
        auto stop = the_clock::now();
        auto diff = stop - start;
        std::cout << diff << std::endl;
    }
    the_clock::time_point start;
};

Now let’s look at different ways to allocate objects. First the normal way:

int normal() {
    vector<Record> v;
    for(int i = 0; i < repetitions; ++i) {
        v.clear();
        for(int j = 0; j < bigBlock; ++j) {
            Record r;
            record_init(r, j);
            v.push_back(r);
        }
    }
    return boost::accumulate(v, 0, [](int i, const Record& r) { return i + r.Id;});
}

Then using a pod_vector (a vector that doesn’t initialize its elements):

int podVector() {
    pod_vector<Record, size> v;
    for(int i = 0; i < repetitions; ++i) {
        v.clear();
        for(int j = 0; j < bigBlock; ++j) {
            Record r;
            record_init(r, j);
            v.push_back(r);
        }
    }
    return boost::accumulate(v, 0, [](int i, const Record& r) { return i + r.Id;});
}

Then using boost pool allocator:

int boostallocator(WhichOne which) {
    boost::pool<> p(sizeof(Record));
    vector<Record, boost::pool_allocator<Record>> v;
    for(int i = 0; i < repetitions; ++i) {
        v.clear();
        for(int j = 0; j < bigBlock; ++j) {
            Record r;
            record_init(r, j);
            v.push_back(r);
        }
    }
    return boost::accumulate(v, 0, [](int i, const Record& r) { return i + r.Id;});
}

Then using shared pointers:

int pointers(WhichOne which) {
    vector<shared_ptr<Record>> v;
    for(int i = 0; i < repetitions; ++i) {
        v.clear();
        for(int j = 0; j < bigBlock; ++j) {
            auto r = make_shared<Record>();
            record_init(*r, j);
            v.push_back(r);
        }
    }
    return boost::accumulate(v, 0, [](int i, const shared_ptr<Record>& r) { return i + r->Id;});
}

Then using Loki smart pointers:

typedef Loki::SmartPtr<Record,
                 Loki::RefCounted,
                 Loki::DisallowConversion,
                 Loki::AssertCheck,
                 Loki::DefaultSPStorage> RecordPtr;

int lokipointers(WhichOne) {
    vector<RecordPtr> v;
    for(int i = 0; i < repetitions; ++i) {
        v.clear();
        for(int j = 0; j < bigBlock; ++j) {
            auto r = RecordPtr(new Record());
            record_init(*r, j);
            v.push_back(r);
        }
    }
    return boost::accumulate(v, 0, [](int i, const RecordPtr& r) { return i + r->Id;});
}

The results are quite interesting. The pod_vector is the fastest, followed by the normal vector, followed by the boost pool allocator, followed by the shared pointers, followed by the Loki pointers.

The pod_vector is faster than the normal vector because it doesn’t initialize its elements. The normal vector is faster than the boost pool allocator because the boost pool allocator has to maintain a pool of memory. The shared pointers are slower than the boost pool allocator because they have to maintain reference counts. The Loki pointers are slower than the shared pointers because they have to maintain reference counts and they have to check for null pointers.

The conclusion is that if you want to write functional code in C++, you should use pod_vector or normal vector. If you need to share objects between different parts of your code, you should use shared pointers. If you need to share objects between different parts of your code and you need to check for null pointers, you should use Loki pointers.

Letter on investing

Thu, 05 Apr 2012 07:57:59 GMT

In 2007 I wrote a blog post on investing. During the last five years, my view evolved a bit. People often ask me how to get a financial education. This is the latest email I sent on the topic.

From: Bolognese, Luca

Sent: 04 April 2012 16:52

To: XXX

Subject: RE: A financial education

Disclaimer: this are just my personal opinions drawn from my 10+ years of investing, reading books and academic papers about it. I can justify most statements below with some academic reference, but that would make this email too long.

Also someone else might read the same material and come up with a different opinion. It is a field that is probabilistic in nature and doesn’t lend itself to certainties. For example, you can make a lot of money out of an investment and still be wrong to have made it. And conversely.

Our brains don’t work well in such fields.

The most important thing in investing is not investing, it is saving the money to invest. If you save enough and are mildly reasonable in your investments (aka you diversify), you are going to be ok. Saving is not mechanically difficult, it is motivationally difficult. The best book I found on the topic is this one. I read earlier editions, this is a new one. I don’t like the investment chapters.
After that, you need to make sure that your financial matters are in order (i.e. you are ensured against catastrophes, you maximize your tax deductions, etc..). This is the field of personal finance. I’ve been suggesting this book to American audiences. It is on Amazon.co.uk, so it might be the UK version.
Now that you have saved money and your finances are in order, you can start thinking about investing. The most important things in investing are: deciding what you believe in, what your risk tolerance is, how important is for your performance to track the market, what your time horizon is and how much time you want to dedicate to investing.
- My risk tolerance is high. I’ve spent a lot of time learning about investing. I’m willing to see the value of my portfolio go down without experiencing emotional distress (I’ve tried it)
- I don’t care about my investment performance tracking the market. I don’t watch financial programs.
- My time horizon is long. These are money I invest for my retirement.
- I’m willing to invest time to keep myself up to date on investment topics and manage my portfolio
- I believe the following (these are the biases that colour my view):
- Diversification among different sources of returns is to be sought aggressively
- Most asset classes are not efficient because of:
  - Predictable flaws in human brain’s processing machinery
  - Institutional constraints on most professional investor (aka they have to track the market they invest in closely)
  - Short term performance incentive for professional investors
- Some asset classes are intrinsically more inefficient than others (i.e. emerging markets micro stocks compared to US large stock) because that’s where the causes of inefficiencies are strongest
- The biggest inefficiencies are among asset classes (i.e. stocks vs bonds vs commodities)
- Most people (myself included) don’t have the time to do the research necessary to invest in individual stocks using the two ways that I believe are ‘right’: quant models or/and fundamental value analysis.
Some books to get you started:
- A good perspective on how to think about the market and the work you need to do if you want to invest in stocks. The intelligent investor.
- Why diversification is important. The intelligent asset allocator. I don’t believe in static asset allocation, but you need to know what it is to believe (or not) in it.
- Moving beyond a static asset allocation with a quant based highly diversified, but simple to implement system. The Ivy Portfolio. I am very tempted to use this one myself
- Why asset classes are often mispriced. Probably the premier quant shop around. Read most quarterly letters.
- Systems to take advantage of such mispricing are described here. I have obtained copies of all the newsletters sent by the author going back to 1991 and backtested some of the systems. These are the system I use. But the newsletter is relatively expensive and complex to follow.
- If you decide to branch out into value analysis of companies. Essays of warren buffett. You would also need to read an introductory and an advanced accounting text. I don’t know UK accounting, so cannot suggest good candidates.
- If you are very conservative and want to just invest in Inflation Linked Bonds (or fearful of markets in general), there is a book for that: here.
With all of that out of the way, here is some practical counsel
- if you have less than 20,000 pounds to invest try to find a fund or etf that invest in a diversified array of asset classes for a small price. I don’t know the UK market, so cannot suggest one.
- If you have more than that and are willing to dedicate a few hours a month, you can start implementing some of the systems in the Ivy Portfolio book. Maybe you integrate it with the GMO 7 years asset class forecasts to push a bit of value bias in your system.
- For most people that’s it. If it becomes a lifelong interest / mania as it is for me, there is a bit more that can be done, but it is unclear to me that the results are necessarily better than the simple systems described above.

Cheers,

.luca

Writing functional code in C++ II – Function composition

Fri, 30 Mar 2012 10:11:00 GMT

Function composition is at the core of functional programming. You start by being very confident that certain very small functions are correct, you compose them in well known ways and you end up being very confident that your final program is correct.

You are very confident that the initial functions are correct because they are very small and side effect free. You are very confident that your program is correct because the means of composition are well known and generate functions that are themselves side effect free.

Such ‘almost primitive’ functions operate on data structures. Each functional language has its own set of data structures and ‘almost primitive’ functions. Probably the most famous ones are contained in the haskell prelude, but F# has similar ones. LINQ in C# is just such a set.

In this article I use the term functional composition in a broad sense as ‘putting functions together’, not in the more technical sense of f(g(x)): dot operator in Haskell or ‘>>’ operator in F#.

Code for this post is here. Thanks to Steve Bower and Andy Sawyer for review and comments.

Here is an example. The F# code below filters in the odd numbers from an array and doubles them. You have two logical operations, filtering and doubling.

let v = [|1;2;3;4;5;6;7;8;9;0|]
let transformed =
    v
    |> Seq.filter(fun i -> i % 2 = 0)
    |> Seq.map ((*) 2)

How would the code look in C++? There are many ways. One is the ‘big fat for loop’, or in C++ 11, for each:

const int tmp[] = {1,2,3,4,5,6,7,8,9,0};
const vector<int> v(&tmp[0], &tmp[10]);
vector<int> filtered;
for(auto i: v)
{
    if(i % 2 == 0)
        filtered.push_back(i * 2);
}

This looks simple enough, apart from the initialization syntax that gets better with C++ 11. But it’s simplicity is misleading. We have now lost the information that our computation is composed by a filter and a map. It’s all intermingled. Obviously, the info is still there in some sense, but you need to read each line in your head and figure out the structure on your own. It is not readily apparent to the reader.

Obviously, things get much worse in a large program composed of several loops where each one does not trivial stuff. As for me, every time I indulge into the ‘big fat for loop pattern’, I end up regretting it either immediately or after some time when I have to make changes to the code. So I try to avoid it.

Wait a minute, you might say, what about STL algorithms? Aren’t you supposed to use those?

The syntax to call them is not very compositional. You cannot take the result of an algorithm and pass it to another easily. It is difficult to chain them that way. Everything takes two iterators, even if 99% of the times you are just using begin and end. Moreover, you have to create all sort of temporary data structures for the sake of not modifying your original one. Once you have done all of that, the syntactic weight of your code overwhelm the simplicity of what you are trying to do.

For example, here is the same code using ‘transform’ for the ‘map’ part of the algorithm:

vector<int> filtered;
copy_if(begin(v), end(v), back_inserter(filtered), [](int x) { return x % 2 == 0;});
vector<int> transformed;
transform(begin(filtered), end(filtered), back_inserter(transformed), [](int x) { return x * 2;});

I’m not arguing that STL is a bad library, I think it is a great one. It’s just that its syntax is not very compositional. And that breaks the magic.

Luckily, help is on the way in the form of the Range library and OvenToBoost. These libraries ‘pack’ the two iterators in a single structure and override the ‘|’ operator to come up with something as below:

auto result =
    v
    | filtered(   [] (int i) { return i % 2 == 0;})
    | transformed([] (int i) { return i * 2;});

If you use Boost lambdas (not sure I’d do that), the syntax gets even better:

auto result =
    v
    | filtered(_1 % 2 == 0)
    | transformed(_1 * 2);

Ok, so we regained compositionality and clarity, but what price are we paying? Apart from the dependency from Boost::Range and OvenToBoost, you might think that this code would be slower than a normal for loop. And you would be right. But maybe not by as much as you would think.

The code for my test is here. I’m testing the perf difference between the following code engineered to put in a bad light the most pleasing syntax. A for loop:

int sum = 0;
for(vector<int>::const_iterator it = v.begin(); it != v.end(); ++it)
    if(*it < 50)
        sum += *it * 2;
return sum;

A Range based code using language lambdas (transformedF is a variation of this):

auto lessThan50 = v | filtered(    [](int i) { return i < 50;})
                    | transformedF([] (int i) { return i * 2;});
return boost::accumulate (lessThan50, 0);

A Range based code using two functors:

auto lessThan50 = v | filtered(Filterer())
                    | transformed(Doubler());
return boost::accumulate (lessThan50, 0);

Where:

struct Doubler {
    typedef int result_type;
    int operator() (int i) const { return i * 2;}
};
struct Filterer {
    typedef int result_type;
    bool operator() (int i) const { return i < 50;}
};

a Range based code using Boost lambdas:

auto lessThan50 = v | filtered(_1 < 50)
                    | transformed(_1 * 2);
return boost::accumulate (lessThan50, 0);

And finally, some code using the STL for_each function:

int sum = 0;
std::for_each( v.cbegin(), v.cend(),
    [&](int i){
        if( i < 50 ) sum += i*2;
    });
return sum;

Notice that I need to run the test an humongous number of times (10000000) on a 100 integers array to start seeing some consistency in results.
And the differences that I see (in nanosecs below) are not huge:

1>                         Language lambda: {1775645516;1778411400;0}
1>                          Functor lambda: {1433377701;1435209200;0}
1>                            Boost lambda: {1327829199;1326008500;0}
1>                                For loop: {1338268336;1341608600;0}
1>                             STL foreach: {1338268336;1341608600;0}

True to be told, by changing the code a bit these numbers vary and, in some configurations, the Range code takes twice as long.

But given how many repetitions of the test I need to run to notice the difference, I’d say that in most scenarios you should be ok.

The ‘|>’ operator over collections is the kind of functional composition, broadly defined, that I use the most, but you certainly can use it over other things as well. You’d like to write something like the below:

BOOST_CHECK_EQUAL(pipe("bobby", strlen, boost::lexical_cast<string, int>), "5");

And you can, once you define the following:

template< class A, class F1, class F2 >
inline auto pipe(const A & a, const F1 & f1, const F2 & f2)
    -> decltype(REMOVE_REF_BUG(f2(f1(a)))) {
    return f2(f1(a));
}

Where REMOVE_REF_BUG is a workaround for a bug in the decltype implementation under msvc (it should be fixed in VS 11):

#ifdef _MSC_VER
    #define REMOVE_REF_BUG(...) remove_reference<decltype(__VA_ARGS__)>::type()
#else
    #define REMOVE_REF_BUG(...) __VA_ARGS__
#endif

I didn’t do the work of creating ‘>>’ , ‘<<’ and ‘<|’ F# operators or wrapping the whole lot with better syntax (i.e. operator overloading?), but it certainly can be done.

Now that we have a good syntax for records and a good syntax for operating over collections, the next instalment will put them together and try to answer the question: how should I create collections of records and operate over them?

Writing functional code in C++ – Records

Fri, 23 Mar 2012 08:11:00 GMT

This is the first of a series of posts about writing functional code in C++. My goal is different from FC++, which is a full fledged ‘environment’ to write functional code. Instead, I want to experiment with some of the new C++ 11 language features and see if one can build reasonably looking functional code and stay pretty close to the language. The idea is to judiciously use macros and external libraries to build a thin layer on top of the language that doesn’t change the performance characteristics of it (aka it doesn’t slow it down) and integrates fine with existing C++ code.

Think of it as an attempt to answer the question: is there a way to write C++ code in a functional style without loosing its ‘C++sness’? We won’t attempt to be as type safe or syntactically pleasing as Haskell or F# , but we’ll attempt to stay as fast and flexible as C++ and make it functional enough to be interesting.

Thanks to Steve Bower and Ganesh Sittampalam for reviewing and to Andy Sawyer for giving me so much feedback that this post can be considered a co-authorship. Code for this post is here.

Let’s first talk about the data types that you typically find in a functional language. Let’s start with Records. They are not part of the functional model per se. You can do without them and just use algebraic data types and tuples. But they are damn convenient, and most functional languages (i.e. Haskell, Clojure, etc…) have them now. We start from them because they map naturally to C++. Records are just like structs, but immutable and having structural equality.

In F# your vanilla record looks like this:

type Person = {
    Name: string
    Id: int
}

Nice and simple. With C++, you become more verbose. A first attempt would be:

struct Person {
    const string Name;
    const int Salary;
};

Which looks nice and easy, but doesn’t quite work because more often than not you need to be able to compare two records for structural equality (which means the value of their fields, not the pointer in memory, defines equality). Records in functional languages automatically support that. But the syntax gets on the ugly side:

struct Person {
    const string Name;
    const int Salary;
    bool operator==(const Person& other) const { return Salary == other.Salary && Name == other.Name;}
    bool operator!=(const Person& other) const { return !(*this == other);}
};

We’ll see how to simplify the syntax later. The syntax on the creation side is not too bad:

Person p = {"Bobby", 2};

Let’s call the above representation, the obvious one. Let’s consider two variations on this scheme. The first one is useful if you want to make your records interoperable with C or with other C++ compilers.

A full discussion of how to achieve these goals would be very long. It will go about discussing what POD types are and how their definition got more precise in C++ 11.

You can look at my experimentations on pod, standard layout and trivially constructible types here. My summary is pretty simple, if you want to achieve all the goals above, you got to use C structs that contain C-compatible types. No const, strings or STL libraries allowed.

The above class would then become:

struct Person {
    char Name[20];
    int Salary;
    bool operator==(const Person& other) const { return Salary == other.Salary && !strcmp(Name,other.Name);}
    bool operator!=(const Person& other) const { return !(*this == other);}
};

Obviously, not being able to use strings or STL collections in your record is a big limitation, but in certain cases you might be able to live with it. Let’s call this representation, the pod one.

You would think you can make the syntax better by doing something like the below:

struct _Person {
    int id;
    char name[20];
    pod_equality(_Person);
};

Where pod_equality is defined as below:

#define pod_equality(Record)                                                                 \
       bool operator==(const Record& other) const {                                          \
              static_assert(std::is_trivial<Record>::value, "Not trivially copyable");       \
              return memcmp(this, &other, sizeof(Record)) == 0;}                             \
       bool operator!=(const Record& other) const { return !(*this == other);}

But you would be wrong (as I was for a while), as comparing memory values doesn’t work in this case because of the padding that the compiler can insert between fields. Such padding can contain random value (whatever was on the stack, for example) which would make the equality return false for structurally equal objects. Also this scheme fails for floating point types (i.e. NaN and signed zeros).

An alternative representation for records, which nicely separates constness from the structure of your record is below. It does have some some advantages that we’ll look at, but in its raw form it is yet more syntax:

namespace Mutable {
    struct Person {
        string Name;
        int Salary;
        bool operator==(const Person& other) const { return Salary == other.Salary && Name == other.Name;}
        bool operator!=(const Person& other) const { return !(*this == other);}
    };
}
typedef const Mutable::Person Person;

Let’s call this representation, the immutable one. Let’s give these guys some usage and talk about their trade-offs. If you want to write a function that increase someone salary, you would write it like this:

template<class T>
T rise_salary1(const T& p) {
    T ret = { p.Name, p.Salary + 1000 };
    return ret;
}

This looks nice and clean, unless your record has a lot of fields. Let me tell you, in real application it probably does. For example:

namespace Mutable {
    struct foo {
        int value1;
        int value2;
        int value3;
        int value4;
        int value5;
        int value7;
        int value6;
        int value8;
        int value9;
    };
}
typedef const Mutable::foo foo;
foo increment_value7( const foo& f )
{
     foo tmp = { f.value1, f.value2, f.value3, f.value4, f.value5, f.value6, f.value7+1, f.value8 };
     return tmp;
}

Thanks to Andy for this example. BTW: did you spot the bug at first sight? What about the other one?

So this syntax is problematic. True to be told, part of the problem is in the sub-optimal initialization syntax in C++. If you could use named parameters, it would be more difficult to introduce bugs, but the syntax would still be ugly. You really need something like the F# syntax:

let r1 = {f = 0.2; k = 3}
let r2 = {r1 with f = 0.1}

Can we do something like that? Well, if we are willing to pass the Mutable one around, we can write this:

template<class T>
T rise_salary2(const T& p) {
    T ret(p);
    ret.Salary += 1000;
    return ret;
}

Or even this:

template<class T>
T rise_salary3(T p) {
    p.Salary += 1000;
    return p;
}

But that doesn’t make us happy, does it? The whole point of making things const is that you want them to be const. If you pass around mutable ones, who knows what’s going to happen inside the method?

There is a middle ground that might be acceptable, which is to write functions so that their interface takes immutable records, but inside the function they operate on mutable ones. This is not a bad pattern in general, as having mutable versions of your immutable records might come useful for optimizing certain algorithms. Luckily the casting rules of C++ favour the bold, so the below works:

Person rise_salary_m(const Person& p) {
    Mutable::Person ret(p);
    ret.Salary += 1000;
    return ret;
}

And doesn’t look too bad either.

Now let’s talk syntax. Defining a record is still a lot of typing (and a lot of reading if you are maintaining the code). F# does it like this:

type Person = {
    Name: string
    Id: int
}

The best I came up with looks like this:

RECORD2(Person,
          string, Name,
          int,    Salary);

And you need a lot of those macros depending on how many fields your record has. You can write this macro to expand to either the Obvious or Immutable representation trivially. It is a bit more complex for the Pod one because of the interesting C++ array declaration syntax with the number of elements after the name of the field.

For the Obvious one it looks like this:

#define RECORD2(n, t1, f1, t2, f2)                                                            \
    struct n {                                                                                \
        const t1 f1;                                                                          \
        const t2 f2;                                                                          \
                                                                                              \
        bool operator==(const n& other) const { return f1 == other.f1 && f2 == other.f2;}     \
        bool operator!=(const n& other) const { return !(*this == other);}                    \
    };

All the usual concerns about macros apply. Moreover all your fields need to have a meaningful == operator.

To summarize, we have found three different representations of records in C++ and tried to alleviate the syntax burden with some macro magick. We’ll go wild in macro-land when we talk about discriminated unions.

A simple scheme to implement Design by Contract in C++

Tue, 06 Mar 2012 16:42:58 GMT

Recently I got interested in C++ again. The new lambda functions in C++ 11 open up a world of opportunities for C++ programmers. I’ll talk more about how you can write functional code in C++ 11 in upcoming posts. For now let’s look at design by contract.

Design by contract is a development style promoted by Bertrand Meyer and it is implemented in his own Eiffel programming language. At core, it advocates using preconditions, postconditions and invariants.

An invariant is an assertion that always holds true for a class after the class has been fully constructed and if the code is not executing inside a method. As a user of the class, you always observe the invariant to be true. As an implementer of the class, you can be assured that the invariant is true before a method is entered and you need to make the invariant true again by the time your method exits.

A preconditions is an assertion that needs to hold true at the start of a function, for the postcondition to be true at the end of it. Taken together, invariant, precondition and postcondition define the contract between the implementer and the user of a class.

Code for this post is here and here. Thanks to Andy Sawyer, Steve Bower and Ganesh Sittampalam for reviewing my code and suggesting improvements.

Preconditions are simple and everyone uses them. They are those little if statements that you put at the start of your functions to make sure that the caller has given you the right parameters.

double divide(double x, double y) {
    if(y == 0) throw new exception("y cannot be 0");
    …
}

These little ‘if’ statements don’t really make the precondition stand out. They can be confused with other, unrelated, ‘if’ statements that do completely different semantic things. A more readable alternative is:

double divide(double x, double y) {
    requires(y != 0);
    …
}

Not an impressive difference, for sure, but kind of nice. The evil macro looks like this:

#ifndef ___PRECOND
#define requires(F) {if((!(F))) throw preexception(__FILE__, __LINE__,"Pre-condition failure: " #F);};
#else
#define requires(F)
#endif

Note that the exception maintains information not just about the file and line number of the failure, but also a textual representation of the failed condition. Such things you can do with macro magick.

Postconditions are trickier. In the case of a side-effect free (pure) function, a postcondition asserts something of interest about the return value. In the case of a class, it asserts something of interest about the state of the class before and after the execution of the method.

Let’s start with a pure function. I like to have all my assertion at the start of the function to allow reasoning about it without looking at implementation details. But that poses the problem that the result is available just at the end of the function. My solution is to enforce this idiom:

double divide(double x, double y) {
    double result;
    requires(y != 0);
    ensures(result < x); // Silly, just to falsify it in tests
    …
    return result;
}

So you need to declare your result upfront. That is the biggest limitation of the overall solution in my opinion. If that is acceptable to you, the trick now is how to execute the postcondition test before the method exits. We can do that by storing a lambda and executing it in the destructor:

typedef std::function<bool ()> ___dbcLambda;
class ___post {
public:
    ___post(const char *file, long line, const char *expr, const ___dbcLambda& postF)
        : _f(postF),
          _file(file),
          _line(line),
          _expr(expr)
    {}
    ~___post()
    {
        if( !std::uncaught_exception() && !_f() )
        {
            throw postexception(_file,_line,_expr);
        }
    }
private:
    const ___dbcLambda _f;
    const char * const _file;
    const long _line;
    const char * const _expr;
};

You might think that you shouldn’t throw exceptions in a destructor. That is something I never understood about the RAII pattern in C++. If I choose to use exceptions as my error notification method, how am I supposed to get notified if there is a problem releasing a resource in RAII, other than by throwing an exception in the destructor?

Maybe because of this, the standard has an uncaught_exception() function that allows you to check if an exception has been thrown, so that you don’t throw another one during stack unwinding. If you really don’t like throwing in the destructor, feel free to assert.

You might be worried about performance, but you really shouldn’t as you can disable all these macros in Release.

The macro then creates a ___post class on the stack.

#define ensures(F) \
    int ___UNIQUE_LINE = __LINE__;  \
    auto ___UNIQUE_POST = ___post( __FILE__, __LINE__, "Post-condition failure:" #F, [&](){return (F);});

The UNIQUE stuff is messy business. Part of it is by design and it is used to make sure that each __post variable has a unique name to have multiple ‘ensures’ in a function. The other part is a workaround for this msvc bug. Let me know if you want more details. I suspect there is a better way to do it.

Here is the full enchilada …

#define ___MERGE(a, b) a##b
#define ___POST(a) ___MERGE(___postcond,a)
#define ___UNIQUE_POST ___POST(__LINE__)
#define ___LINE(a) ___MERGE(___line, a)
#define ___UNIQUE_LINE ___LINE(__LINE__)

The case in which a postcondition is used inside a method of a class is even trickier because the postcondition must be able to compare the state of the class at the entrance of the method to the state of the class at its exit. Assuming a Counter object with an Add method and assuming ‘___pre’ captures the state of the counter at the start of the method, you’d like to write something like:

void Add(int x) {
    ensuresClass(this->c_ == ___pre.c_ + x);
    …
}

Now, this is tricky. The only way to capture the ‘old’ state in ‘___pre’ is by making a copy of it and store it there. This is what the code below does:

#define ensuresClass(F) \
    auto ___pre(*this); \
    auto ___UNIQUE_POST = ___post( __FILE__, __LINE__, "Post-condition failure: " #F, [&](){return (F);});

More troubling is the possibility that the class doesn’t have a copy constructor. In that case you explicitly need to associate a value with ‘___pre2’ by passing it as the first parameter to the appropriate macro as in the code below:

void Add(int x) {
    ensuresClass2(this->c_, c_ == ___pre2 + x);
}

Which is implemented as follows:

#define ensuresClass2(ASS,F) \
    auto ___pre2(ASS); \
    auto ___UNIQUE_POST = ___post( __FILE__, __LINE__, "Post-condition failure: " #ASS " is ___pre2 in " #F, [&](){return (F);});

And I know about the giant ass …

Now for invariants. The user should implement an isValid() method on his class as below:

bool isValid() { return c_ >= 0;}

Then he should add an ‘invariant()’ call at the start of each method, at the end of each constructor and at the start of each destructor:

void Add(int x) {
    invariant();
    requires(x < 10);
    ensures(this->c_ == ___pre.c_ + x);
    …
}

This calls the ‘isValid’ function at the start of the method and at the end of it using the same destructor trick:

#define invariant() \
    if(!(this->isValid())) throw preexception(__FILE__, __LINE__,"Invariant failure"); \
    auto ___UNIQUE_INV = ___post( __FILE__, __LINE__, "Invariant failure", [&](){return this->isValid();});

All the above machinery is not at all equivalent to having such constructs in the language, but it is simple enough and with a decent enough syntax to be interesting.

Now a caveat: I have no idea if any of this works. It does work in my examples and its behaviour seems reasonably safe to me, but I haven’t tried it out on any big codebase and haven’t stressed it enough for me to be confident recommending its usage. So, use it at your own risk, let me know how it goes.

Adventure in parserland – parsing lambda expressions in F# – Part V

Fri, 16 Sep 2011 06:57:00 GMT

We are now going to look at a solution which is concise, efficient and gives sophisticated error messages. It is also less than 20 lines of code. We’ll be using FParsec.

FParsec is a port of an Haskell library. It is a parser combinator library or, as I like to think of it, an internal DSL to build parsers in F#. My usual disclaimer: I’m not an expert in FParsec. It is likely that, if you are an expert, you can come up with more maintainable/efficient/elegant version of this parser.

The whole code is below:

let ws = " \t\n"
let specialChars = ".)(\\\n"
let pWs = spaces
let pName = manyChars (noneOf (ws + specialChars)) |>> EName
let pExpr, pExprRef = createParserForwardedToRef<Expression, Unit>()
let curry2 f a b = f(a,b)
let pFunction = pchar '\\' >>. pipe2 pName (pchar '.' >>. pExpr) (curry2 Function)
let pApplication = pchar '(' >>. pipe2 pExpr (pWs >>. pExpr) (curry2 Application)
                          .>> pWs .>> pchar ')'
do pExprRef := pFunction <|> pApplication <|> pName
let pExpressions = sepBy pExpr spaces1

This mirrors pretty closely the grammar we are trying to parse. A program is a bunch of expressions separated by whitespaces.

let pExpressions = sepBy pExpr spaces1

sepBy is a combinator that takes a parser that defines what to parse and a parser that defines what the separator should be. And the separator should be one or more spaces …

pExpr is either a function, an application or a name. the operator <|> is a choice combinator that tries each parser in order. It tries the right parser just if the left parser fails and it doesn’t consume input. So it doesn’t backtrack. If you need a backtracking parser, you’ll need to get acquainted with the attempt combinator.

do pExprRef := pFunction <|> pApplication <|> pName

A function starts with a ‘', then comes a name, a dot and an expression.

let pFunction = pchar '\\' >>. pipe2 pName (pchar '.' >>. pExpr) (curry2 Function)

I know that might look crazy (and maybe it is), but just bear with me. Someone , who I’m not sure I can name, once told me that functional programming is great to write code, but terrible to read it and debug it. The phrase stayed with me as containing a grain of truth. In any case, in the code above:

. is a combinator that says “use the left parser, discard its value and then use the right one, returning its value”. Try to guess what .>> does …
pipe2 is a combinator that says “apply the first parser, the second parser and then call a function passing as parameters the values returned by the two parsers
curry2 is a function combinator that transform a function that takes a tuple to a function that takes the parameters as untupled

let curry2 f a b = f(a,b)

An application works similarly, but differently …

let pApplication = pchar '(' >>. pipe2 pExpr (pWs >>. pExpr) (curry2 Application)
                          .>> pWs .>> pchar ')'

The only difference is that now we have to consume the optional whitespaces and the ‘)’ at the end. A rule of thumb that I use is to use >>. to flow the result through and pipeX when I need more than one result.

The last thing is pName, which consume chars until it finds either a whitespace or a special char.

let ws = " \t\n"
let specialChars = ".)(\\\n"
let pWs = spaces
let pName = manyChars (noneOf (ws + specialChars)) |>> EName

And there you have it, a lexer, a parser all in 20 lines of code. I don’t like the code that I wrote above much. I’m sure I could refine it plenty and it probably contains some bugs, but it gives an idea of what is possible with FParsec.

Adventure in parserland – parsing lambda expressions in F# – Part IV

Fri, 09 Sep 2011 06:09:00 GMT

Let’ now look at the parser. First let’s review the grammar:

(*
        <expression> ::= <name> | <function> | <application>
        <name> ::= nonblank character sequence
        <function> ::= \ <name> . <body>
        <body> ::= <expression>
        <application> ::= ( <function expression> <argument expression> )
        <function expression> ::= <expression>
        <argument expression> ::= <expression>
    *)

And the data type to represent it:

type Name = string
and Body = Expression
and Function = Name * Expression
and FunctionExpression = Expression
and ArgumentExpression = Expression
and Expression =
| EName of string
| Function of Expression * Body
| Application of FunctionExpression * ArgumentExpression
| EOT

In essence, the data type need to store all the information needed for subsequent stages of computation (i.e. beta reductions and such). The closer it is to the grammar, the better. In this case it looks pretty close.

Remember what is the main goal of our parser:

let parseTextReader: TextReader -> seq<Expression> =
                    textReaderToLazyList >> tokenStream >> parseExpressions

We have already looked at TextReaderToLazyList and tokenStream. Now it is the time to look at parseExpressions. It’s goal is to parse the LazyList and return a sequence of expressions. The choice of returning a sequence at this point is to make the parseTextReader, which is the main function in the program, return a more ‘standard’ type.

and parseExpressions tokens = seq {
   let tokens = parseOptionalWs tokens
   let expr, tokens = parseExpr tokens
   let tokens = parseOptionalWs tokens
   match expr with
    | EOT   -> yield EOT
    | exp   -> yield exp; yield! parseExpressions tokens }

parseOtionalWs simply skips ahead whatever whitespaces it finds.

and parseOptionalWs tokens = match tokens with
                                | LazyList.Nil -> LazyList.empty
                                | LazyList.Cons(h, t) ->
                                    match h with
                                       | Ws _ -> parseOptionalWs t
                                       | _ -> tokens

parseExpr is more interesting. It is the main switch that creates expression kinds.

let rec parseExpr tokens = match tokens with
                            | LazyList.Nil -> EOT, LazyList.empty
                            | LazyList.Cons(h, t) ->
                                match h with
                                    | EOF -> parseEOF tokens
                                    | Name _ -> parseName  tokens
                                    | Lambda -> parseFunction tokens
                                    | OpenParens -> parseApplication tokens
                                    | token -> errorAtStart "Expression" token

parseEOF is not.

and parseEOF tokens = EOT, LazyList.empty

parseName just returns a EName, unwrapping it from Name.

and parseName tokens = EName (head tokens |> unwrapName), tail tokens

Unwrap just unwraps it.

let unwrapName = function
    | Name(s) -> s
    | tok -> errorExpecting "a Name" <| writeToken tok

parseFunction just conumes a Lambda, a name, a Dot token, a body (i.e. \x.x)and assembles them in a Function:

and parseFunction tokens =
    let tokens = consumeToken Lambda tokens
    let name, tokens = parseName tokens
    let tokens = consumeToken Dot tokens
    let body, tokens = parseExpr tokens
    Function(name, body), tokens

consumeToken tries to consume a token generating an error if it doesn’t find it:

let consumeToken token =
    genericConsumeToken (fun token' _ -> errorExpecting (writeToken token') (writeToken token)) token

genericConsumeToken is just a generalization of the function above:

let genericConsumeToken noMatch token = function
    | LazyList.Nil -> LazyList.empty
    | LazyList.Cons(h, t) as originalTokens ->
        match h with
        | tok when tok = token -> t
        | tok -> noMatch token originalTokens

The last thing left to consume is an application which is in this form (func args):

and parseApplication tokens =
    let tokens = consumeToken OpenParens tokens
    let funExpr, tokens = parseExpr tokens
    let tokens = parseOptionalWs tokens
    let argExpr, tokens = parseExpr tokens
    let tokens = consumeToken CloseParens tokens
    Application(funExpr, argExpr), tokens

Various error and utility functions are defined below:

let errorEOF expecting = failwith  ("Expected " + expecting + ", got EOF")
let errorExpecting expecting gotToken = failwith ("Expected " + expecting + ", got" + gotToken)
let errorAtStart expecting gotToken = failwith ("Expected " + expecting + " which cannot start with" + writeToken gotToken)
let tail = LazyList.tail
let head = LazyList.head

And that is the parser. All 100+ lines of it. As you can tell it is rather formulaic to go from a grammar to a lexer and a parser, which is why you shouldn’t do it, but instead let a tool generate the code for you given the grammar or use FParsec.

We have written 200+ code and I don’t think we can be too proud of our achievement. It is:

Certainly buggy
Primitive in error handling
Not tail recursive (big text is likely to blow up our stack)
Probably inefficient

So let’s look next at a better way to do it.

Adventure in parserland – parsing lambda expressions in F# – Part III

Fri, 02 Sep 2011 15:25:00 GMT

Let’s start from the lexer. Remember, I wrote this code based on my memory of how a lexer ought to look like. I didn’t read again the relevant chapters in the Dragon book. But I think it came out all right after all.

The tokenStream function we looked at last time takes a LazyList and returns a LazyList. It uses the unfold method on LazyList to call matchToken on each char until the stream is empty.

let rec tokenStream chars =
    LazyList.unfold
        (fun chList ->
            match chList with
            | LazyList.Nil -> None
            | chList ->
                let token, chList' = matchToken chList
                Some(token, chList')
        )
        chars

A token is what gets passed up to the parser to do syntactic analysis on. It is the vocabulary of our language. The lexer divide a phrase in words, the parser put together the words in a phrase. So, these are the words.

type Token =
    | Name of string
    | Dot
    | OpenParens
    | CloseParens
    | Lambda
    | Def
    | Ws of string
    | NewLine
    | EOF

Matching is a process whereas you try to return the token that you have read plus the list of characters yet to be read. Matching a Token is defined below:

let matchToken = function
    | LazyList.Nil                 -> EOF, LazyList.empty
    | LazyList.Cons(h, t) as chars ->
        match h with
        | ch when isWs ch -> matchWs chars
        | ch when isSpecialChar ch -> matchSpecialChar ch t
        | _ -> matchString chars

A token is either nothing, a whitespace, a special char or anything else.

Let’s look at what matching each one of them means. Matching whitespaces means consuming them and remembering what was consumed.

let matchWs chars =
    let value, remainingChars = matchSeriesOfChars isWs chars
    Ws value, remainingChars

matchSeriesOfChars takes a predicate and a LazyList of chars and returns the string composed of all the consecutive chars for which the predicate is true, plus, as always, the remaining chars to be matched. In this case the predicate returns true if the char is a whitespace.

To write matchSeriesOfChars I need a function that reverses a LazyList. Not having found such thing, I wrote it.

let reversell l =
    let rec go l' a = match l', a with
                        | LazyList.Nil, a -> a
                        | LazyList.Cons(h, t), a -> go t (LazyList.cons h a)
    go l LazyList.empty

Then I wrote matchSeriesOfChars. The function uses an accumulator. It adds to the front whenever the predicate is true, it reverses it and translates it to a string (I could have reversed the string instead, it might have been better).

let matchSeriesOfChars comparer chars =
    let rec go result = function
        | LazyList.Nil    -> charListToString(reversell result), LazyList.empty
        | LazyList.Cons(h, t) -> if comparer h then go (LazyList.cons h result) t
                                 else charListToString (reversell result), LazyList.cons h t
    go LazyList.empty chars

These are predicates we’ll use later on to recognize characters:

let isInString (ch: char) (s: string) = s.IndexOf(ch) <> -1
let isWs (chr: char) = isInString chr wsChars
let isNameChar (chr: char) = not (isInString chr (wsChars + specialChars))
let isSpecialChar ch = isInString ch specialChars

wsChar and specialChars are defined below:

let wsChars = " \t"

let charTokens =
    Map.ofList [
        '.', Dot
        '(', OpenParens
        ')', CloseParens
        '\\', Lambda
        '\n', NewLine
     ]
let specialChars = charTokens |> Map.fold (fun s k v -> s + k.ToString()) ""

Getting back to the more important matching functions, matching a special character is defined as a simple lookup in the charToken map:

let matchSpecialChar ch chars = Map.find ch charTokens, chars

We are left with matchString, this simply matches the characters until it finds a char that cannot be part of a name. It then looks it up in a list of special strings. If it finds it, it returns it, otherwise it just returns the name.

let stringTokens =
    Map.ofList [
        "Def", Def
    ]

let matchString chars =
    let value, remainingChars = matchSeriesOfChars isNameChar chars
    let specialString = Map.tryFind value stringTokens
    if specialString.IsSome
        then specialString.Value, remainingChars
        else Name(value), remainingChars

And we are done with the lexer, all of 100+ lines of it …

Adventure in parserland – parsing lambda expressions in F# – Part II

Fri, 26 Aug 2011 14:25:00 GMT

The parser starts simple with the following two functions to parse either a string or a file. I use the XXX_Readers_ because I want to lazy read character by character.

let parseString s =
    let reader = new StringReader(s)
    parseTextReader reader
let parseFile fileName =
    let reader = new StreamReader(fileName: string)
    parseTextReader reader

The whole parser is in the following two lines:

let parseTextReader: TextReader -> seq<Expression> =
                    textReaderToLazyList >> tokenStream >> parseExpressions

I need to specify the signature otherwise the compiler gets confused : wait, does it take a StringReader or a StreamReader? You better tell me!

The function is a composite of three functions applied in sequence:

Translate a TextReader to a LazyList
Translate a LazyList to a LazyList (lexer)
Translate a LazyList to a LazyList (parser)

My usage of LazyList as the workhorse for the program is because I want to match on the head of the stream of chars/tokens in a lazy way.

I love it when a program naturally decomposes in such simple understandable pieces. I impute some of that to functional programming. For one reason or another, in my 15+ years of object oriented programming, I’ve rarely got to the core of a problem with such immediacy.

A sequence of operations likes the above would be lost in a protected overridden implementation of a base class somewhere (or something else equally long to pronounce). The beauty would be lost somewhere in the vast machinery required to support it.

In any case, TextReaderToLazyList is a trivial generator function that uses the unfold function of LazyList to read a character at the time.

let textReaderToLazyList textReader = LazyList.unfold (fun (ts:TextReader) ->
    let ch = ts.Read()
    if ch = -1 then None else Some(char ch, ts)) textReader

The next step is to look at either the lexer, going bottom up, or the parser, going top down.

Adventure in parserland – parsing lambda expressions in F# – Part I

Fri, 19 Aug 2011 06:53:00 GMT

This is part of my ‘things that I do in the empty spaces between one meeting and the next one, which might end up being vaguely interesting’. It is a lambda expression parser.

The full source code is here.

I actually have two versions of it: one written longhand and the other one written with FParsec. Just to be clear: I’m no expert of either.

And just to be more clear: I think writing most parsers longhand in the way I am about to show is crazy. You either use FParsec or fslex / fsyacc.

I have a strong distaste for additional compilation steps. I think it lingers on from MFC project types of 15/20 years ago. I was one of these crazy folks that would generate the project, wrap the generated code (with some generalizations) in my own library and use that one from then on.

So I prefer FParsec. I’m ok rewriting left recursive rules and its performance has never been a problem for me. Here is a table that compares the different approaches.

But I started wondering about coding a a recursive descent parser for a simple grammar by hand, fully knowing the foolishness of the idea. Thanks to Jose for code reviewing it.

The inspiration for the grammar comes from this book.

(*
        <expression> ::= <name> | <function> | <application>
        <name> ::= nonblank character sequence
        <function> ::= \ <name> . <body>
        <body> ::= <expression>
        <application> ::= ( <function expression> <argument expression> )
        <function expression> ::= <expression>
        <argument expression> ::= <expression>
    *)

In English, an expression is either a name, a function or an application. A name is a bunch of characters (better defined in the code). A function is ‘', a name, ‘.’ and an expression. An application is ‘(‘, an expression, whitespaces, an expression and ‘)’.

Some testcases for the above grammar and the parsers written to parse it are below. It should be intuitive what this code does just by the name of the functions. Even it isn’t, check that the expressions symbol contains valid productions from the grammar above.

module Test
open Microsoft.FSharp.Collections
open Xunit
open LambdaEngine
open Parser
open Lexer
open FParser

let writeTokenStream stream = Seq.fold (fun acc token -> acc + writeToken token) "" stream

let rec writeExpr = function
        | EName(s) -> s
        | Function(expr, body) -> writeToken Lambda + writeExpr expr + writeToken Dot + writeExpr body
        | Application(funExpr, argExpr) -> writeToken OpenParens + writeExpr funExpr + writeToken (Ws(" "))
                                            + writeExpr argExpr + writeToken CloseParens
        | EOT -> ""

let tokenStreams = [
    ""
    "(\xs.xs \y.(y \x.y))"
    "(\xst.xst \y.(y  \x.y))"
    " "
    "x"
    "(x y)"
    ]

let expressions = [
    ""
    "(\x.x \y.(y \x.y))"
    "x"
    "(x y)"
    ]

let stringToCharList s =
    let textReader = new System.IO.StringReader(s)
    textReaderToLazyList textReader

[<Fact>]
let testTokenizer () =
    let testTokenStream s =
        let stream = tokenStream <| stringToCharList s
        let s1 = writeTokenStream stream
        Assert.Equal(s, s1)
    tokenStreams |> List.iter testTokenStream

let testExpr parseFunction s =
    let exprs = parseFunction s
    let s1 = exprs |> Seq.fold (fun s expr -> s + writeExpr expr) ""
    Assert.Equal(s, s1)

[<Fact>]
let testParser () = expressions |> List.iter (testExpr parseString)

[<Fact>]
let testFParser () = expressions |> List.iter (testExpr fparseString)

In the next instalment, we’ll start looking at the real code for the parser.

Write Yourself a Scheme in 48 Hours in F# – Part VII

Fri, 12 Aug 2011 07:11:00 GMT

Let’s talk about the environment now. This is the part of the interpreter that I like the least. It is a global variable and it contains a list of (string, LispVal) where the LispVal is mutable.

type Env = (string * LispVal ref) list ref

This is pretty bad. First of all, it immediately cuts off any option of running interpreters in different threads. Moreover, it makes a lot of functions in the evaluator to have side effects. That makes it much harder to reason about them.

In a world where I am provided with infinite time and energy, I would change it. In this world, I won’t. If you try your hand at doing it, make sure that you pass all the testcases before declaring victory. The scope rules of Scheme are not all that obvious. A code reviewer called them the Italian scoping rules because he thought I got them wrong …

In any case, there isn’t much to the symbol table management. You can create an empty one:

let nullEnv (): Env = ref List.empty

Check if a variable is bound:

let keyEq name (k, _) = name = k
let isBound var (env: Env) = !env |> List.exists (keyEq var)

Get a variable out:

let getVar var (env: Env) =
    let result = !env |> List.tryFind (keyEq var)
    match result with
    | None -> throw (UnboundVar("Getting an unbound variable: " , var))
    | Some(_, r) -> !r

Set the value of an existing variable:

let setVar var value (env:Env) =
    let result = !env |> List.tryFind (keyEq var)
    match result with
    | Some(_, v) -> v := value ; value
    | None -> throw (UnboundVar("Setting an unbound variable: " , var))

Or define a new variable in the environment. Note that if the variable already exist, its value gets set.

let define (env:Env) var value =
    let result = !env |> List.tryFind (keyEq var)
    match result with
    | Some(_, v) -> v := value ; value
    | None ->
        env := [var, ref value] @ !env; value

You can also bind a list of (string, LispVal) to the environment by prepending it to the existing ones:

let bindVars bindings (env:Env) =
   ref ((bindings |> List.map (fun (n, v) -> n , ref v)) @ !env)

Once you accept the evil of the global mutable variable scheme, these functions are easy enough.

The only piece left is error management. This is where my implementation differs from the Haskell version the most. In essence, I throw exception and catch them to report errors, while the Haskell version uses a monad to propagate the error information.

I have a LispError that represents everything that can go wrong:

type LispError =
    | NumArgs of int * LispVal list
    | TypeMismatch of string * LispVal
    | ParseError of string * FParsec.Error.ParserError
    | BadSpecialForm of string * LispVal
    | NotFunction of string * string
    | UnboundVar of string * string
    | Default of string
    | IOError of string

I wrap it in an exception:

exception LispException of LispError

This is what I throw in various places in the code.

let throw le = raise (LispException(le))

I then catch it at the outer layer:

let evalString env expr =
    try
        expr |> readExpr |> eval env
    with
    | LispException(error) -> String (showError error)

And display the error by using the below function:

let showError = function
    | NumArgs(expected, found) -> "Expected " + expected.ToString() + " args; found values " + unwordsList found
    | TypeMismatch(expected, found) -> "Invalid type: expected " + expected + ", found " + showVal found
    | ParseError(msg, _) -> "Parse Errror" + msg
    | BadSpecialForm(message, form) -> message + showVal form
    | NotFunction(message, func) -> message + func
    | UnboundVar(message, varName) -> message + varName
    | Default(message) -> message
    | IOError(message) -> message

And that’s all there is to it. I hope you guys and gals enjoyed this seven part extravagance. Cheers.

Write Yourself a Scheme in 48 Hours in F# – Part VI

Fri, 05 Aug 2011 07:08:00 GMT

The evaluator takes as an input a LispVal. Where does it come from? There must be something that converts your textual input into it. That is the job of the parser.

I have used FParsec to build my parser. FParsec is a fantastic library to build parsers. It is a perfect showcase of the composition potential that functional code yields.

When you write an FParsec parser you compose many little parsers to create the one parser that works for your language. The resulting code looks very much like your language grammar, but you don’t need a separate code generation compilation step to produce it.

There is one element of ugliness in the syntax to create recursive parsers. You need to define two global variables that can be referred to before they are constructed. This is an artefact of how F# works. So you need a line in your code that looks like this:

let parseExpr, parseExprRef : LispParser * LispParser ref = createParserForwardedToRef()

With that piece of machinery out of the way, we can focus on the parser itself. Our goal here is to parse expressions and generate LispVal. We need a LispParser like the below (the second generic parameter is for advanced usage).

type LispParser = Parser<LispVal, unit>

We need to parse all the kind of expressions that the user can type. Notice in the below the use of a computation expression to simplify the syntax. Also note that lists and dotted lists look very much the same until you encounter the ‘.’ character. You could disambiguate the situation by extracting out the commonality in a separate kind of expression. I decided instead to instruct the parser to backtrack if it gets it wrong (attempt). This is slower, but keeps the code identical to our conceptual model. I value that greatly.

do parseExprRef := parseAtom
                   <|> parseString
                   <|> parseNumber
                   <|> parseQuoted
                   <|> parse {
                           do! chr '('
                           let! x = (attempt parseList) <|> parseDottedList
                           do! chr ')'
                           return x
                       }

Let’s start from the top. Parsing an atom means parsing something that starts with a letter or symbol and continues with letters, symbols or digits. Also “#t” and “#f” can be resolved at parsing time.

let parseAtom : LispParser = parse {
        let! first = letter <|> symbol
        let! rest = manyChars (letter <|> symbol <|> digit)
        return match first.ToString() + rest with
               | "#t" -> Bool true
               | "#f" -> Bool false
               | atom -> Atom atom
}

A string is just a bunch of chars (except ‘') surrounded by ’ ” ‘.

let parseString : LispParser = parse {
    do! chr '"'
    let! xs = manyChars (noneOf "\"")
    do! chr '"'
    return String(xs)
}

A number is just one or more digits. I am afraid we just support integers at this stage …

let parseNumber : LispParser = many1Chars digit |>> (System.Int32.Parse >> Number)

A quoted expression is jut a ‘' followed by an expression.

let parseQuoted : LispParser = chr '\'' >>. parseExpr |>> fun expr -> List [Atom "quote"; expr]

A list is just a bunch of expressions separate by at least one space.

let parseList : LispParser = sepBy parseExpr spaces1 |>> List

A dotted list starts in the same way (hence the backtracking above), but then has a dot, one or more spaces and an expression.

let parseDottedList : LispParser = parse {
    let! head = endBy parseExpr spaces1
    let! tail = chr '.' >>. spaces1 >>. parseExpr
    return DottedList (head, tail)
}

And here are a bunch of functions used throughout the code, presented here for completeness.

let spaces1 : LispParser<unit> = skipMany1 whitespace
    let chr c = skipChar c
    let endBy  p sep = many  (p .>> sep)
    let symbol : LispParser<char> = anyOf "!$%&|*+-/:<=?>@^_~#"

This is all the code you need to translate text to a LispVal to feed the evaluator. That is pretty impressive.

There is also a function to go the other way, from a LispVal to text. It is used in implementing the testcases and to print out diagnostics.

```fsharp let rec showVal = function | String contents -> “"” + contents + “"” | Atom name -> name | Number num -> num.ToString() | Bool t -> if t then “#t” else “#f” | List l -> “(” + unwordsList l + “)” | DottedList (head, tail) -> “(” + unwordsList head + ” . ” + showVal tail + “)” | PrimitiveFunc() -> “” | Port () -> “” | Func({ parms = parms; varargs = varargs; body = body; closure = closure }) -> “(lambda (” + unwordsList (parms |> List.map (String)) + (match varargs with | None -> “” | Some(arg) -> ” . ” + arg) + “) …)” and unwordsList = List.map showVal >> String.concat ” “

Write Yourself a Scheme in 48 Hours in F# – Part V

Fri, 29 Jul 2011 07:26:00 GMT

We have one loose end to tie in the evaluator: the primitive operators. These are things that the interpreter knows intrinsically. There is a list of them below.

let rec primitives =
     [
        "+",    numericBinop (+)
        "-",    numericBinop (-)
        "*",    numericBinop (*)
        "/",    numericBinop (/)
        "mod",  numericBinop (%)
        "=",    numBoolBinop (=)
        "<",    numBoolBinop (<)
        ">",    numBoolBinop (>)
        "/=",   numBoolBinop (<>)
        ">=",   numBoolBinop (>=)
        "<=",   numBoolBinop (<=)
        "&&",   boolBoolBinop (&&)
        "||",   boolBoolBinop (||)
        "string=?",     strBoolBinop (=)
        "string>?",      strBoolBinop (>)
        "string<?",      strBoolBinop (<)
        "string<=?",    strBoolBinop (<=)
        "string>=?",    strBoolBinop (>=)
        "car",  car
        "cdr",  cdr
        "cons", cons
        "eq?", eqv
        "eqv?", eqv
        "equal?", equal
        // IO primitives
        "apply", applyProc
        "open-input-file", makePort FileAccess.Read
        "open-output-file", makePort FileAccess.Write
        "close-input-port", closePort
        "close-output-port", closePort
        "read", readProc
        "write", writeProc
        "read-contents", readContents
        "read-all", readAll
     ]

Having seen the above list, it now becomes clearer why the primitiveBindings function was defined as such. It just binds these pairs into the environment.

let primitiveBindings () =
    (nullEnv ()) |> bindVars [ for v, f in primitives -> v, PrimitiveFunc f ]

numericBinop unpacks the numbers, applies the provided operator and packs the result back in the Number.

let numericBinop op parms =
    if List.length parms < 2
        then throw <| NumArgs(2, parms)
        else parms |> List.map unpackNum |> foldl1 op |> Number

While we are at it, we can define _fold1 (_it tends to be useful)

let foldl1 op = function
    | h::t -> List.fold op h t
    | [] -> throw (Default("Expected a not empty list, got an empty list"))

The other XBinops work similarly …

let boolBinop unpacker op args =
    match args with
    | [ left; right ] -> Bool (op (unpacker left) (unpacker right))
    | _ -> throw (NumArgs(2,args))
let numBoolBinop = boolBinop unpackNum
let strBoolBinop = boolBinop unpackStr
let boolBoolBinop = boolBinop unpackBool

We now have to look at the family of unpackers. They all work rather similarly. Notice Scheme making an effort to get a number out of a string and to get anything out of a list. Strong type folks won’t like that. Oh well, just remove these lines …

let rec unpackNum = function
    | Number n  -> n
    | String n  -> let success, result = System.Int32.TryParse n
                   if success
                       then result
                       else throw (TypeMismatch("number", String n))
    | List [n]  -> unpackNum n
    | notNumber -> throw (TypeMismatch("number", notNumber))
let rec unpackStr = function
    | String s -> s
    | Number n -> n.ToString()
    | Bool b   -> b.ToString()
    | List [s]  -> unpackStr s
    | noString -> throw (TypeMismatch("string", noString))
let rec unpackBool = function
    | Bool b -> b
    | List [b]  -> unpackBool b
    | noBool -> throw (TypeMismatch("boolean", noBool))

Now back to the list of primitive operators, there are the signature LISP operators car, cdr and cons. Just understanding the first line for each function should be enough to get an idea of what they do.

let car = function
    | [List (x :: _)] -> x
    | [DottedList (x :: _, _)] -> x
    | [badArg] -> throw (TypeMismatch("pair", badArg))
    | badArgList -> throw (NumArgs(1, badArgList))
let cdr = function
    | [List (x :: xs)] -> List xs
    | [DottedList ([xs], x)] -> x
    | [DottedList ((_ :: xs), x)] -> DottedList (xs, x)
    | [badArg] -> throw (TypeMismatch("pair", badArg))
    | badArgList -> throw (NumArgs(1, badArgList))
let cons = function
    | [x; List xs] -> List (x :: xs)
    | [x; DottedList (xs, xlast)] -> DottedList (x :: xs, xlast)
    | [x1; x2] -> DottedList([x1], x2)
    | badArgList -> throw (NumArgs(2, badArgList))

We then need to work our way to implement eqv (aka eq? in Scheme). We first define a function that tests that two LispVal are the same. It should be pretty self explanatory (the list piece is kind of cute).

let rec eqvPrim e1 e2 =
        match e1, e2 with
        | (Bool b1, Bool b2) -> b1 = b2
        | (Number n1, Number n2) -> n1 = n2
        | (String s1, String s2) -> s1 = s2
        | (Atom a1, Atom a2) -> a1 = a2
        | (DottedList (xs, x), DottedList(ys, y)) -> eqvPrim (List (xs @ [x])) (List (ys @ [y]))
        | (List l1, List l2) -> l1.Length = l2.Length && List.forall2 eqvPrim l1 l2
        | _ -> false

Now we wrap the result in a Bool. Doing it this way avoid repeating the wrapping in each single line of eqvPrim (thanks to Tobias for spotting this refactoring).

let eqv = function
          | [e1; e2] -> Bool (eqvPrim e1 e2)
          | badArgList -> throw (NumArgs (2, badArgList))

Equal? checks if there is any unpacking scheme that can be used to test equality of the two elements of a two element list.

let equal = function
    | [arg1; arg2] ->
        let unpackEqual = numUnpackEq arg1 arg2 ||
                          strUnpackEq arg1 arg2 ||
                          boolUnpackEq arg1 arg2
        Bool (eqvPrim arg1 arg2 || unpackEqual)
    | argsList -> throw (NumArgs(2, argsList))

We need to define equality of packed primitive types. We do it nicely below.

let tryUnpacker (unpack : LispVal -> 'a) (op : 'a -> 'a -> bool) arg1 arg2 =
    try op (unpack arg1) (unpack arg2) with _ -> false
let numUnpackEq = tryUnpacker unpackNum (=)
let strUnpackEq = tryUnpacker unpackStr (=)
let boolUnpackEq = tryUnpacker unpackBool (=)

The apply statement maps more or less directly to our apply function.

applyProc = function
            | [func; List args] -> apply func args
            | func :: args -> apply func args
            | [] -> throw (Default("Expecting a function, got an empty list"))

And we are left with the I/O processing functions. We are simply wrapping a FileStream in a Port.

let makePort fileAccess = fileIOFunction (fun fileName ->
                                File.Open(fileName,FileMode.OpenOrCreate, fileAccess) |> Port)
    let closePort = function
                    | [Port(port)] -> port.Close() ; Bool true
                    | _ -> Bool false

We then can read and write from it. Notice how the lack of arguments makes us do it from the standard Console.

let rec readProc port =
    let parseReader (reader:TextReader) = reader.ReadLine() |> readExpr
    match port with
       | [] -> parseReader(System.Console.In)
       | [Port(port)] ->
            use reader = new StreamReader(port)
            parseReader (reader)
       | args -> throw (NumArgs(1, args))
let writeProc objPort =
    let write obj (writer: TextWriter) = writer.Write(showVal obj) ; Bool true
    match objPort with
    | [obj] -> write obj (System.Console.Out)
    | [obj ; Port(port)] ->
        use writer = new StreamWriter(port)
        write obj writer
    | args -> throw (NumArgs(1, args))

There you go. A full evaluator in two blog posts!! Next up, the parser.

Write Yourself a Scheme in 48 Hours in F# – Part IV

Fri, 22 Jul 2011 07:52:00 GMT

It is the evaluator turn. It is a big file, let’s see if I can fit it in a single post.

Aptly enough, the most important function is called eval.

eval env = function
| String _ as v -> v
| Number _ as v -> v
| Bool _ as v -> v
| Atom var -> getVar var env
| List [Atom "quote"; v] -> v
| List [Atom "if"; pred; conseq; alt] -> evalIf env pred conseq alt
| List [Atom "load"; fileName] -> load [fileName] |> List.map (eval env) |> last
| List [Atom "set!" ; Atom var ; form] -> env |> setVar var (eval env form)
| List [Atom "define"; Atom var; form] -> define env var (eval env form)
| List (Atom "define" :: (List (Atom var :: parms) :: body)) ->
    makeNormalFunc env parms body |> define env var
| List (Atom "define" :: (DottedList ((Atom var :: parms), varargs) :: body)) ->
    makeVarargs varargs env parms body |> define env var
| List (Atom "lambda" :: (List parms :: body)) -> makeNormalFunc env parms body
| List (Atom "lambda" :: (DottedList(parms, varargs) :: body)) -> makeVarargs varargs env parms body
| List (Atom "lambda" :: ((Atom _) as varargs :: body)) -> makeVarargs varargs env [] body
| List (func :: args) ->
    let f = eval env func
    let argVals = List.map (eval env) args
    apply f argVals
| badForm -> throw (BadSpecialForm("Unrecognized special form", badForm))

This is the core of the evaluator. It takes as an input the LispVal generated by the parser and an environment and returns a LispVal that is the result of the reduction. As a side effect, it occasionally modify the environment. I carefully crafted the previous phrase to maximize the discomfort of the functional programmers tuned in. Such fun 🙂

More seriously (?), here is what it does:

If it is a String, Number of Bool, just return it
If it is an Atom, return its value
If it is a quote statement, return what is quoted (read your little schemer manual)
If it is an if statement, evaluate it (see below)
If it is a load statement, load the file (see below) and evaluate each expression using the current environment. Return the last expression in the file
If it is a set!, set the value of the passed variable to the evaluated form
If it is a define, do almost the same as above (except that you don’t throw if the variable doesn’t exist, but you create it)
If it is a define that defines a function (it has that shape), create a ‘function slot’ in the environment. That is a (functionName, FuncRecord) pair (see below)
If it is a lambda, return the FuncRecord that describe the inline function
If it is a function call, evaluate the expression that describe the function (yes, you can do that in Lisp!!), evaluate the arguments, and apply the function to the arguments
Otherwise, it must be a bad form, throw it back to the calling function to do something meaningful with it

We have a bunch of ‘see below’ to take care of. We’ll look at them in order.

First the ‘if’ statement. If the evaluated predicate is Bool(True) evaluate the consequent, otherwise evaluate the alternative. For some reason, I wrote it the other way around.

and
    // 1a. If the evaluation of the pred is false evaluate alt, else evaluate cons
    evalIf env pred conseq alt =
        match eval env pred with
        | Bool(false) -> eval env alt
        | _ -> eval env conseq

Then there is the load function. It reads all the test and gets out the list of LispVal contained in it.

let load = fileIOFunction (fun fileName -> File.ReadAllText(fileName)
                                           |> readExprList)

ReadExprList is part of the parser. We’ll look at it then. Sufficient here to say that it takes a string and returns a list of LispVal.

FileIOFunction just encapsulates a common pattern in all the file access functions. I don’t like such mechanical factorization of methods, without any real reusability outside the immediate surroundings of the code. But I like repetition even less.

let fileIOFunction func = function
    | [String fileName] -> func (fileName)
    | [] -> throw (IOError("No file name"))
    | args -> throw (NumArgs(1, args))

A family of functions create FuncRecord given appropriate parameters. I seem to have lost memory of the contortions related to the last one. If I end up having to work again on this code, I’ll need to figure it out again. Note to myself, please comment this kind of code next time.

let makeFunc varargs env parms body =
            Func ({parms = (List.map showVal parms); varargs = varargs; body = body; closure = env})
let makeNormalFunc = makeFunc None
let makeVarargs = showVal >> Some >> makeFunc

apply is the other workhorse function in the evaluator. The best way to understand it is to start from the bottom (where bindVars starts the line). We are binding the arguments and the variable arguments in the closure that has been passed in. We then evaluate the body. But the body is just a list of LispVal, so we just need to evaluate them in sequence and return the result of the last one.

and apply func args =
    match func with
    | PrimitiveFunc(f) -> f args
    | Func ({parms = parms; varargs = varargs; body = body; closure = closure}) ->
        let invalidNonVarargs = args.Length <> parms.Length && varargs.IsNone
        let invalidVarargs = args.Length < parms.Length && varargs.IsSome
        if invalidVarargs || invalidNonVarargs
        then
            throw (NumArgs(parms.Length, args))
        else
            let remainingArgs = args |> Seq.skip parms.Length |> Seq.toList
            let evalBody env = body |> List.map (eval env) |> last
            let rec zip xs1 xs2 acc =
                match xs1, xs2 with
                | x1::xs1, x2::xs2 -> zip xs1 xs2 ((x1, x2)::acc)
                | _ -> acc
            let bindVarArgs arg env =
                match arg with
                | Some(argName) -> bindVars [argName, (List remainingArgs)] env
                | None -> env
            bindVars (zip parms args []) closure
                |> bindVarArgs varargs
                |> evalBody
    | funcName -> throw (NotFunction("Expecting a function, getting ", showVal funcName))

This is enough for one post. Next time we’ll finish the evaluator.

Write Yourself a Scheme in 48 Hours in F# – Part III

Fri, 15 Jul 2011 07:12:00 GMT

Very often my code ends up having the following form: parse an input to create an intermediate data structure and evaluate the structure to produce an output. Strangely, many years ago, when my code was object oriented, that wasn’t the case. Or at least I wasn’t explicitly aware of it.

When you write an interpreter or a compiler, things always work out like that, but I see the same pattern in almost everything I produce: from financial backtesting to chart libraries. Sometimes when, out of laziness or stupidity, I forego the intermediate structure, I end up in the end having to retrofit it in. Simply processing input and generating output at the same time rarely cuts it. But it is tempting because you get going pretty fast and I’m tricked into it occasionally.

Hence the first thing that I find myself reasoning about is often the particular form of such intermediate structure. In this case it looks like the following:

type Env = (string * LispVal ref) list ref
and FuncRecord = { parms: string list; varargs: string option; body: LispVal list; closure: Env}
and LispVal =
    | Atom of string
    | List of LispVal list
    | DottedList of LispVal list * LispVal
    | Number of int
    | String of string
    | Bool of bool
    | PrimitiveFunc of (LispVal list -> LispVal)
    | Func of FuncRecord
    | Port of System.IO.FileStream

This LispVal structure has one constructor for each kind of expression (production) that is allowed in Scheme. Or at least that ones I support …

It is important that each one stores all the information that is necessary for the evaluator to evaluate the expression. No more, no less. Here is a brief description:

Atom: it is a kind of a constant in Scheme. This is probably the worst definition ever given for it. Please read about it in your little schemer book.
List: is the main Scheme type. It represents a list of expressions.
DottedList: this is the bizarre Scheme way to pass optional parameters
Number: is a number 🙂 You will discover which kind of number when we talk about the parser
String : is a string
Bool: #t, #f
PrimitiveFunc: is the representation for the primitive operators/functions that are burned into the interpreter. It is just a function that takes a list of LispVal and returns a LispVal
Func: is a user defined function. Notice that the body of it is simply a list of LispVal. This is why LISP is so powerful. Also notice that a closure gets passed to it for the ‘captured’ variables.
Port: is a slightly goofy representation of an in/out stream
Anything else (i.e. macros) is not supported, but this would be the first piece to change if they were.

The only remaining code to address is:

type Env = (string * LispVal ref) list ref

This is the symbol table and it is ugly. It is not multithread safe either. But it works and it is close enough to the Haskell version so I decided to retain it. A proper code review would ‘strongly suggest’ rewriting the code to pass it around to each function instead of using ‘ref’ or using the state monad encapsulated in a computation expression. Any of these solutions is left as an exercise to the reader (use the testcases to validate that you get it right).

We could go in many different direction from here. We could talk about:

The evaluator
The parser
The symbol table
Error handling

To keep with the top – down approach I’ve been espousing. I’ll probably talk about the evaluator next.

Write Yourself a Scheme in 48 Hours in F# – Part II

Fri, 08 Jul 2011 07:01:00 GMT

Usually, when I do blog posts that are all about code, I write them ‘bottom up’. I start talking about the most primitive types and functions and build up from there toward higher abstractions. I think this is a pretty common way of doing it.

For this series I’m going to try the opposite. I start with the code that creates the REPL window and move down from there toward the guts of the interpreter. I hold the opinion that, if the code is written right, this should be ok. The naming scheme and general structure of it should allow understanding it at different levels.

Or at least I hope so.

Let’s start from the main function. Depending on the number of arguments it either runs the REPL window or executes whatever is in the file passed in as the first argument using the other arguments as parameters.

[<EntryPoint>]
let main(args: string[]) =
    match Array.toList args with
    | [] -> runRepl ()
    | filename :: args -> runOne filename args
    0

The latter case is coded in the below function. It first load all the primitive operators (i.e. ‘+’, ‘-’ etc…) and the standard library. The word ‘load’ above is a little misleading. In reality it adds them to the environment. It then proceeds to add the arguments that were passed on. As the last step, it evaluates the ‘load’ command by using the newly created environment, it transforms the returned token to a string and prints it.

let runOne (filename : string) (args : list<string>) =
    let env = primitiveBindings ()
                |> loadStdLib
                |> bindVars [ "args", List (List.map String args) ]
    List [Atom "load"; String filename] |> eval env |> showVal |> printStr

Running the REPL windows is equally simple. Load the primitive operators and the standard library, show a prompt and evaluate the input until the input is ‘Quit’.

let runRepl () =
    let env = primitiveBindings () |> loadStdLib
    until (fun s -> s = "Quit" || s = "quit") (fun () -> readPrompt "Lisp>>> ") (evalAndPrint env)

readPrompt is pretty simple:

let printStr (s: string) = Console.Write(s)
let readPrompt (s: string) = printStr s ; Console.ReadLine ()

EvalAndPrint is written as a chain of functions (lookup the ‘>>’ operator in F#) and just evaluate the string, transform the result to a string, prints it and newline it.

let newLine () = Console.WriteLine()
let evalAndPrint env = evalString env >> showVal >> printStr >> newLine

evalString parses the string and evaluates the expression. Note the exception management. This is a result of my decision of throwing an exception every time something goes wrong instead of using a monad to pass the state around. I think it is pretty clear, but haskellers might disagre. This is one of the main differences from the Haskell version.

let evalString env expr =
    try
        expr |> readExpr |> eval env
    with
    | LispException(error) -> String (showError error)

For the sake of completeness, here is until. Maybe there is a library function somewhere that I could have used?

let rec until pred prompter evaluator =
    let result = prompter ()
    if not (pred result) then
        evaluator result
        until pred prompter evaluator

Back on the main flow of the code, loadStdLib just loads the standard file and returns the populated environment.

let loadStdLib env =
    eval env (List [Atom "load"; String "stdlib.scm"]) |> ignore
    env

primitiveBindings creates a new empty environment and adds a bunch of pairs (primitiveName, LispVal –> LispVal). LispVal is a representation of a Scheme expression, so the second part of the tuple is simply a reduction from one expression to another (hopefully simpler in some sense). We’ll talk more about LispVal in upcoming posts.

let primitiveBindings () =
    (nullEnv ()) |> bindVars [ for v, f in primitives -> v, PrimitiveFunc f ]

There you have it. That’s the full implementation for the REPL window. Next post, we’ll look at LispEval and the evaluator.

Write Yourself a Scheme in 48 Hours in F# – Part I

Thu, 30 Jun 2011 15:41:12 GMT

Hi, I’m back. I’ve finally sorted out the guidelines for blogging in Credit Suisse.

Here is something I have been playing around with in the spare time between one meeting and the next one. It is a Scheme interpreter that includes a REPL window. The full code is here.

All the smarts for it come from this Wiki Book. I just ported the code to F# (and modified it a bit). I thought the comparison might be interesting, so here we go. Thanks to Tobias and Jose for reviewing the code, find one bug and suggest improvements.

Before we start looking at the real code, here is what we are trying to accomplish in form of test cases. If you are a bit rusty on LISP syntax, you might want to try and see if you understand what it does.

Our goal is to make all this XUnit test cases pass. Each of the lists below contains the Scheme statement and the result to display in the REPL window.

```fsharp open Xunit open Lisp.Repl open Lisp.Parser open Lisp.SymbolTable let eval env = evalString env >> showVal let initEnv () = primitiveBindings () |> loadStdLib let test tests = let env = initEnv () tests |> List.iter (fun (expr, result) -> Assert.Equal(result, eval env expr)) [] let simpleEval() = let tests = [ “(+ 2 2)”, “4” “(+ 2 (- 4 1))”, “5” “(- (+ 4 6 3) 3 5 2)”, “3” ] test tests [] let errorCheck() = let tests = [ “(+ 2 "two")”, “"Invalid type: expected number, found "two""” “(+ 2)”, “"Expected 2 args; found values 2"” “(what? 2)”, “"Getting an unbound variable: what?"” ] test tests [] let moreEval() = let tests = [ “(< 2 3)”, “#t” “(> 2 3)”, “#f” “(>= 3 3)”, “#t” “(string=? "test" "test")”, “#t” “(string=? "abcd" "dsft")”, “#f” “(if (> 2 3) "no" "yes")”, “"yes"” “(if (= 3 3) (+ 2 3 (- 5 1)) "unequal")”, “9” “(cdr ‘(a simple test))”, “(simple test)” “(car (cdr ‘(a simple test)))”, “simple” “(car ‘((this is) a test))”, “(this is)” “(cons ‘(this is) ‘test)”, “((this is) . test)” “(cons ‘(this is) ‘())”, “((this is))” “(eqv? 1 3)”, “#f” “(eqv? 3 3)”, “#t” “(eqv? ‘atom ‘atom)”, “#t” ] test tests [] let assignement() = let tests = [ “(define x 3)”, “3” “(+ x 2)”, “5” “(+ y 2)”, “"Getting an unbound variable: y"” “(define y 5)”, “5” “(+ x (- y 2))”, “6” “(define str "A string")”, “"A string"” “(< str "The string")”, “"Invalid type: expected number, found "A string""” “(string<? str "The string")”, “#t” ] test tests [] let closure() = let tests = [ “(define (f x y) (+ x y))”, “(lambda ("x" "y") …)” “(f 1 2)”, “3” “(f 1 2 3)”, “"Expected 2 args; found values 1 2 3"” “(define (factorial x) (if (= x 1) 1 ( x (factorial (- x 1)))))”, “(lambda ("x") …)” “(factorial 10)”, “3628800” “(define (counter inc) (lambda (x) (set! inc (+ x inc)) inc))”, “(lambda ("inc") …)” “(define my-count (counter 5))”, “(lambda ("x") …)” “(my-count 3)”, “8” “(my-count 6)”, “14” “(my-count 5)”, “19” ] test tests [] let predefinedFunctions() = let tests = [ “(map (curry + 2) ‘(1 2 3 4))”, “(3 4 5 6)” “(filter even? ‘(1 2 3 4))”, “(2 4)” ] test tests [] let varargsCountCheck() = let tests = [ “(define (sum x y . lst) (fold + ( x y) lst))”, “(lambda ("x" "y" . lst) …)” “(sum 1 2 3)”, “5” “(sum 1 1 1)”, “2” “(sum 1 2)”, “2” “(sum 1)”, “"Expected 2 args; found values 1"” ] test tests

I have been DEEP FRIED!

Wed, 31 Mar 2010 20:50:32 GMT

Yep, here is my interview with the Deep Fried Bytes guys on various software topics. Enjoy.

Tracking spread trades in F# (and WPF MVVM) – Part II

Fri, 19 Mar 2010 23:33:00 GMT

I wanted to experiment with MVVM and WPF in F#, so I decided to create a little graphical interface for the csv file that drives the spread tracking application. When I started I thought I needed some kind of a grid with Submit/Cancel buttons, but the more I thought about it, the more I realized that I wouldn’t need them.

See, I’ve always be one to complain about our current paradigm of Open File / Close File / Save File arguing that the user shouldn’t know about an entity called ‘file’. He shouldn’t be exposed to the fact that the application is just an in-memory copy of an hard disk artifact. His mental model should simply be: I open a document, I work on it, I close it, if needed I can make a copy; if I have problems I can revert to a previous version of the same document; If I make an error I can use ‘undo’ to revert it. There are no files/save/submit/cancel in such paradigm. There is no file system.

On the technical side I wanted to experiment with MVVM, even if in this case, the paradigm is overkilled (can really use this word?), given the simplicity of the application.

In any case, the ViewModel is in F#. It uses two utility classes:

// TODO: refactor to remove code repetition below
[<AbstractClass>]
type ViewModelBase () =
    let propertyChanged = new Event<PropertyChangedEventHandler, PropertyChangedEventArgs>()
    interface INotifyPropertyChanged with
        [<CLIEvent>]
        member this.PropertyChanged = propertyChanged.Publish
    member internal this.RaisePropertyChangedEvent(propertyName:string) =
        if not(propertyName = null) then
            let e = new PropertyChangedEventArgs(propertyName)
            let i = this :> INotifyPropertyChanged
            propertyChanged.Trigger(this, e)
type ObservableCollectionWithChanges<'a when 'a :> INotifyPropertyChanged> () =
    inherit ObservableCollection<'a> ()
    let propertyChanged = new Event<PropertyChangedEventHandler, PropertyChangedEventArgs>()
    member c.PropertyChanged = propertyChanged.Publish
    member private c.RaisePropertyChangedEvent(propertyName:string) =
        if not(propertyName = null) then
            let e = new PropertyChangedEventArgs(propertyName)
            let i = c :> INotifyPropertyChanged
            propertyChanged.Trigger(c, e)
    member c.Add(o) =
        base.Add(o)
        o.PropertyChanged.Add(fun x -> c.RaisePropertyChangedEvent(""))

The first one is used as a base for all the viewmodel entities in the application, the second one serves as the base for all the collections. They both define the customary PropertyChanged event. The latter adds itself as an observer to each object added to the collection so that, whenever one changes, it gets notified and can notify its own observers. Look at the c.Add method. A lot of repetitive code here, I would heed the advice of the comment on top if this were production code.

Each line in the csv file is represented as a ResultViewModel, hence the following:

type ResultViewModel (d:DateTime, sLong, sShort, tStop) =
    inherit ViewModelBase ()
    let mutable date = d
    let mutable stockLong = sLong
    let mutable stockShort = sShort
    let mutable trailingStop = tStop
    new () = new ResultViewModel(DateTime.Today, "", "", 0)
    member r.Date with get() = date
                       and set newValue =
                            date <- newValue
                            base.RaisePropertyChangedEvent("Date")
    member r.StockLong with get() = stockLong
                       and set newValue =
                            stockLong <- newValue
                            base.RaisePropertyChangedEvent("StockLong")
    member r.StockShort with get() = stockShort
                        and set newValue =
                            stockShort <- newValue
                            base.RaisePropertyChangedEvent("StockShort")
    member r.TrailingStop with get() =
                                trailingStop
                          and set newValue =
                                trailingStop <- newValue
                                base.RaisePropertyChangedEvent("TrailingStop")
    member r.IsThereAnError = r.TrailingStop < 0 || r.TrailingStop > 100

I need the empty constructor to be able to hook up to the DataGrid add-new capability. There might be an event I could use instead, but this is simple enough (even if a bit goofy).

The main view model class then looks like the following:

type MainViewModel (fileName:string) as self =
    inherit ViewModelBase ()
    let mutable results = new ObservableCollectionWithChanges<ResultViewModel>()
    let loadResults () =
        parseFile fileName
        |> Array.iter (fun (d,sl, ss, ts) ->
                        results.Add(new ResultViewModel(d, sl, ss, ts)))
    do
        loadResults ()
        results.CollectionChanged.Add(fun e -> self.WriteResults())
        results.PropertyChanged.Add(fun e -> self.WriteResults())
    member m.Results with get() = results
                     and set newValue =
                        results <- newValue
                        base.RaisePropertyChangedEvent("Results")
    member m.WriteResults () =
        let rs = results
                 |> Seq.map (fun r -> r.Date, r.StockLong, r.StockShort, r.TrailingStop)
        let thereAreErrors = results |> Seq.exists (fun r -> r.IsThereAnError)
        if not thereAreErrors then
            writeFile fileName rs

Things here are more interesting. First of all, in the constructor I load the results calling my model (which I created in Part I of this series). I then subscribe to both the events fired by the collection of results. The former is triggered when an object is added/removed, the latter is triggered when an object changes one of its properties. When one of them fires, I simply write the new state back to the file. This allows me to get rid of Submit/Cancel buttons. What the user sees on the screen is synchronized with the disk at all times. The user doesn’t need to know about the file system.

If this were real, I would also implement an undo/redo mechanism. In such case, my reliance on object events might be unwise. I would probably route all the user changes through a command mechanism, so that they can be undo more easily.

That’s it for the modelview. The View itself is as follows:

<Window x:Class="SpreadTradingWPF.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:spreadTrading="clr-namespace:SpreadTradingWPF"
        Title="Spread Trading" Height="350" Width="525" SizeToContent="WidthAndHeight">
        <Window.Resources>
            <spreadTrading:DateToShortStringConverter x:Key="DateToShortStringC" />
            <LinearGradientBrush x:Key="BlueLightGradientBrush" StartPoint="0,0" EndPoint="0,1">
                    <GradientStop Offset="0" Color="#FFEAF3FF"/>
                    <GradientStop Offset="0.654" Color="#FFC0DEFF"/>
                    <GradientStop Offset="1" Color="#FFC0D9FB"/>
            </LinearGradientBrush>
            <Style TargetType="{x:Type DataGrid}">
                <Setter Property="Margin" Value="5" />
                <Setter Property="Background" Value="{StaticResource BlueLightGradientBrush}" />
                <Setter Property="BorderBrush" Value="#FFA6CCF2" />
                <Setter Property="RowBackground" Value="White" />
                <Setter Property="AlternatingRowBackground" Value="#FDFFD0" />
                <Setter Property="HorizontalGridLinesBrush" Value="Transparent" />
                <Setter Property="VerticalGridLinesBrush" Value="#FFD3D0" />
                <Setter Property="RowHeaderWidth" Value="20" />
            </Style>
    </Window.Resources>
        <StackPanel HorizontalAlignment="Center" Name="stackPanel1" VerticalAlignment="Top" Margin="20">
            <TextBlock Text="Spread Trading" Width="135" HorizontalAlignment="Center" FontSize="18" FontWeight="Bold" FontStretch="ExtraExpanded" />
            <DataGrid Height="Auto" Width="Auto" Margin="5" ItemsSource="{Binding Results}" CanUserAddRows ="True" CanUserDeleteRows="True" AutoGenerateColumns="False">
                <DataGrid.RowValidationRules>
                    <spreadTrading:ResultValidationRule ValidationStep="UpdatedValue"/>
                </DataGrid.RowValidationRules>
                <DataGrid.RowValidationErrorTemplate>
                    <ControlTemplate>
                        <Grid Margin="0,-2,0,-2"
                              ToolTip="{Binding RelativeSource={RelativeSource
                              FindAncestor, AncestorType={x:Type DataGridRow}},
                              Path=(Validation.Errors)[].ErrorContent}">
                            <Ellipse StrokeThickness="0" Fill="Red"
                                Width="{TemplateBinding FontSize}"
                                Height="{TemplateBinding FontSize}" />
                            <TextBlock Text="!" FontSize="{TemplateBinding FontSize}"
                                FontWeight="Bold" Foreground="White"
                                HorizontalAlignment="Center"  />
                        </Grid>
                    </ControlTemplate>
                </DataGrid.RowValidationErrorTemplate>
            <DataGrid.Columns>
                    <DataGridTextColumn Header="Date" Binding="{Binding Date, Converter= {StaticResource DateToShortStringC}}"  IsReadOnly="false"/>
                    <DataGridTextColumn Header="Long" Binding="{Binding StockLong}"/>
                    <DataGridTextColumn Header="Short" Binding="{Binding StockShort}" />
                    <DataGridTextColumn Header="Stop" Binding="{Binding TrailingStop}" />
            </DataGrid.Columns>
            </DataGrid>
        </StackPanel>
</Window>

Notice that I styled the grid and I used the right incantations to get validation errors and to bind things properly. The DateToShortString converter used for the date field might be mildly interesting. It’s in the Utilities.cs file together with a validation rule that just delegates to the IsThereAnError method on each entity. In a bigger application, you could write this code in a much more reusable way.

[ValueConversion(typeof (DateTime), typeof (string))]
public class DateToShortStringConverter : IValueConverter
{
public Object Convert(
        Object value,
        Type targetType,
        Object parameter,
        CultureInfo culture)
{
    var date = (DateTime) value;
    return date.ToShortDateString();
}
public object ConvertBack(
    object value,
    Type targetType,
    object parameter,
    CultureInfo culture)
    {
        string strValue = value as string;
        DateTime resultDateTime;
        if (DateTime.TryParse(strValue, out resultDateTime))
        {
            return resultDateTime;
        }
        return DependencyProperty.UnsetValue;
    }
}
public class ResultValidationRule : ValidationRule
{
    public override ValidationResult Validate(object value, CultureInfo cultureInfo)
    {
        var result = (value as BindingGroup).Items[0] as ResultViewModel;
        if(result.IsThereAnError)
            return new ValidationResult(false, "TrailingStop must be between 0 and 100");
        else
            return ValidationResult.ValidResult;
    }
}

After all these niceties, this is what you get.

Tracking spread trades in F# (and hooking up XUnit and FsCheck) – Part 1

Sat, 13 Mar 2010 01:18:37 GMT

I have a bunch of spread trades open. Spread trades are trades where you buy something and you sell something else generally in the same amount. You hope to profit from the widening of the spread between the price of the two instruments.

I place stop loss orders or trailing stops for all my trades. I have various tool that automatically notify me when a stop loss or trailing stop is hit. For spread trades I don’t have such a tool, hence I decided to build it.

I defined maximum adverse excursion for a spread trade as the percentage difference between the current value of ‘long price’ / ‘short price’ and its maximum value from the point the trade was placed (aka Current(‘long price’ / ‘short price’) / max (‘long price’ / ‘short price’) – 1 ). This goes from 0 to 100. If the maximum adverse excursion is larger than the trailing stop (using closing prices only), then I want to be notified by email.

I decided to create a simple exe and use Task Scheduler to run it at the end of the trading day. The program reads a file with all the open spread trades, downloads their prices, calculates maximum adverse excursion and sends an email if it is larger than the trailing stop. I also built a little WPF veneer to manipulate the configuration file.

Here is what my common.fs file looks like.

namespace Spread
module internal Common =
    open System
    let internal isValidDate s =
        let v, _ = DateTime.TryParse(s)
        v
    let internal isValidTrailingStop s =
        let v1, n = Int32.TryParse(s)
        if not(v1) then
            false
        else
            n >= 0 && n <= 100
    let internal isValidTicker (t:string) = not(t.Contains(","))
    let internal isValidLine (l:string) = l.Split([|','|]).Length = 4
    let internal elseThrow message expression = if not(expression) then raise message
    let internal elseThrowi i message expression = if not(expression) then failwith (sprintf "On line %i : %s" i message)

Notice the isValidTicker function. Yep, I’m using a CSV file to store the list of spread trades. Also I often end up using the little elseThrow functions that I originally used in the Excel functions library to check preconditions.

Here is an example of using them for the parseLine function:

// parse a line in the csv config file, assumes valid csv, dates and trailing stop in [0,100]
let internal parseLine lineNumber line =
    isValidLine line                |> elseThrowi lineNumber "badly formatted line"
    let values = line.Split([|','|])
    isValidDate values.[0]          |> elseThrowi lineNumber "badly formatted date"
    isValidTicker values.[1]        |> elseThrowi lineNumber "long ticker has a comma in it"
    isValidTicker values.[2]        |> elseThrowi lineNumber "short ticker has a comma in it"
    isValidTrailingStop values.[3]  |> elseThrowi lineNumber "trailing stop has to be between 0 and 100 included"
    DateTime.Parse(values.[0]), values.[1].Trim(), values.[2].Trim(), int values.[3]

As you can see, the csv format is (dateOfTrade, longTicker, shortTicker, trailingStop). Let’s now look and the FsCheck testcase for this function.

let writeLine (date:DateTime) (tickerLong:string) (tickerShort:string) (trailingStopValue:int) =
    sprintf "%s,%s,%s,%i" (date.ToShortDateString()) tickerLong tickerShort trailingStopValue
[<Fact;Category("Fast Tests")>]
let can_parse_valid_lines () =
    let  prop_parseLine (lineNumber:int) date tickerLong tickerShort trailingStopValue =
        let line = writeLine date tickerLong tickerShort trailingStopValue
        let values = line.Split([|','|])
        (isValidLine(line) && isValidDate values.[0] && isValidTicker values.[1] && isValidTicker values.[2]
                                                                                        && isValidTrailingStop values.[3])
            ==> lazy
                let actual = parseLine lineNumber line
                (date, tickerLong.Trim(), tickerShort.Trim(), trailingStopValue) = actual
    check config prop_parseLine

In FsCheck you state properties of your functions and FsCheck generates random values to test them. In this case I’m asserting that, given a date, tickerLong, tickerShort, trailingStopValue, I can write them to a string, read them back and I get the same values. Frankly, I was skeptical on the utility of such exercise, but I was wrong. That’s how I discovered that tickers cannot have commas in them (among other things).

To hook up FsCheck and XUnit (aka to run FsCheck property checking as normal testcases), you need to write the below black magic code.

let xUnitRunner =
    { new IRunner with
        member x.OnArguments(_,_,_) = ()
        member x.OnShrink(_,_) = ()
        member x.OnFinished(name, result) =
            match result with
                | True data -> Assert.True(true)
                | _ -> failwith (testFinishedToString name result)
    }
let config = {quick with Runner = xUnitRunner}

Also, to run XUnit with your brand new .net 4.0 project, you need to add xunit.gui.exe.config to the XUnit directory with the following content:

While we are talking about such trivialities, I compile my testcases as executable, so that I can easily run them under debug. I also add the InternalsVisibleTo attribute, so that I can test internal stuff. Many of my algorithms are in internal functions and I want to test them in isolation.

[<assembly:InternalsVisibleTo("SpreadTrackingTests")>]
    do

Given the previous function, I can then parse text and files with the following:

let internal parseText (lines:string) = lines.Trim().Split([|'\n'|]) |> Array.mapi parseLine
let public parseFile fileName = File.ReadAllText fileName |> parseText

I need to load closing prices. I’m using my own library to load prices. That library is pretty badly designed. Also, the function below should be factorized in several sub-functions. It kind of shows how you can write spaghetti code in a beautiful functional language as F# if you really try hard. But let’s not worry about such subtleties for now …

let internal loadClosingPrices (endDate:DateTime) tickersStartDate  =
    // format parameters to conform to loadTickersAsync
    let tickersLong, tickersShort =
        tickersStartDate
        |> Array.map (fun (startDate:DateTime, ticker1:string, ticker2:string, _) ->
                (ticker1, {Start = startDate; End = endDate}), (ticker2, {Start = startDate; End = endDate}))
        |> Array.unzip
    let prices = tickersShort
                 |> Array.append tickersLong
                 |> Array.toList
                 |> loadTickersAsync
                 |> Async.RunSynchronously
                 |> Array.map (fun (ticker, span, obs) -> ticker, obs (*|> asHappened 1. |> adjusted adjStart*))
    let len = tickersLong.Length
    let longObs = Array.sub prices 0 len
    let shortObs = Array.sub prices len len
    // removes divs and splits
    let choosePrices observation = match observation.Event with Price(pr) -> Some(observation) | _ -> None
    let combineOverTickerObservations f tickerObservations =
        tickerObservations
        |> Array.map (fun (ticker, observations) ->
                                            ticker,
                                            observations |> List.choose f |> List.rev)
    let longPrices = combineOverTickerObservations choosePrices longObs
    let shortPrices = combineOverTickerObservations choosePrices shortObs
    longPrices, shortPrices

In the above, tickerStartDate is an array of (trade date * long ticker * short ticker * trailingStop) which is what is produced by our parseLine function. The function first separates out long tickers from short ones.

let tickersLong, tickersShort =
    tickersStartDate
    |> Array.map (fun (startDate:DateTime, ticker1:string, ticker2:string, _) ->
            (ticker1, {Start = startDate; End = endDate}), (ticker2, {Start = startDate; End = endDate}))
    |> Array.unzip

It then puts them together again in a single Array, to be able to pass it to the loadTickerAsync functions. It runs the function, waits for the results and then returns an array of (ticker * observations).

let prices = tickersShort
             |> Array.append tickersLong
             |> Array.toList
             |> loadTickersAsync
             |> Async.RunSynchronously
             |> Array.map (fun (ticker, span, obs) -> ticker, obs |> asHappened 1. |> adjusted adjStart)

The data is downloaded as it comes from Yahoo, which is a mix of adjusted and not adjusted data. asHappened transforms it so that everything is as it really happened, adjusted then adjusts it for the effect of dividends and splits. Think of this two function as ‘make the data right’.

We then split them again to get the long and short series. The point of merging them and splitting them is to call loadTickersAsync just once instead of twice. There are better ways to do it.

let len = tickersLong.Length
        let longObs = Array.sub prices 0 len
        let shortObs = Array.sub prices len len

At this point we remove the observations that represents dividends or splits, as we are interested just in prices and we return the resulting observations.

let choosePrices observation = match observation.Event with Price(pr) -> Some(observation) | _ -> None
let combineOverTickerObservations f tickerObservations =
    tickerObservations
    |> Array.map (fun (ticker, observations) ->
                                        ticker,
                                        observations |> List.choose f |> List.rev)
let longPrices = combineOverTickerObservations choosePrices longObs
let shortPrices = combineOverTickerObservations choosePrices shortObs
longPrices, shortPrices

The List.rev at the end is interesting. Somewhere in the loadTickerAsync/asHappened/adjusted triad of functions I end up reversing the list. I should fix the bug instead of workaround it, but this is just a blog post, not production code, so I’ll let it slip.

Now that we have our price observations, we need to extract the price values and calculate the sequence of ratios (long price / short price).

let internal calcRatioSeries longPrices shortPrices =
    let extractPrice obs = match obs.Event with Price(pr) -> pr.Close | _ -> failwith "At this point divs and splits should have been removed"
    let longValues = longPrices |>  List.map extractPrice
    let shortValues = shortPrices |> List.map extractPrice
    shortValues |> List.map2 (/) longValues

Having this ratio series, we can calculate the maximum adverse excursion, incorrectly called trailing stop below.

let internal calcTrailingStop ratioSeries = List.head ratioSeries / List.max ratioSeries - 1.

We then create a function that puts it all together.

type public Result = {RatioName:string; CurrentTrailing:int; TrailingStop:int} with
    override x.ToString() = x.RatioName + "\t\t" + x.CurrentTrailing.ToString() + "\t\t" + x.TrailingStop.ToString()
// reads a csv file (startDate, longTicker, shortTicker, trailingStop) and returns an array of results
let public processFile fileName endDate =
    let fileInfo = parseFile fileName
    let longPrices, shortPrices = loadClosingPrices endDate fileInfo
    let ratioSeries = Array.map2 (fun l s -> fst l + "/" + fst s, calcRatioSeries (snd l) (snd s)) longPrices shortPrices
    ratioSeries |> Array.mapi (fun i (name, series) ->
                    let (_,_,_,ts) = fileInfo.[i]
                    {RatioName = name; CurrentTrailing = - int (Math.Round (calcTrailingStop series * 100., 0));
                                                                                                       TrailingStop = ts})

The function takes a fileName and an endDate, the latter parameter is for the sake of testcases that has to work in the past, so that the data doesn’t change on them.

Now we need to send an email. The code below works for me:

let sendEmail smtpServer port fromField toField subject body (user:string) (password:string) =
    let client = new SmtpClient(smtpServer, port)
    client.Credentials <- new NetworkCredential(user, password)
    client.EnableSsl <- true
    client.Send(fromField, toField, subject, body)
// gets the password from a file under C: so that when I post it on my blog I don't forget to delete it
let getPassword () =
    File.ReadAllText(@"D:\Documents and Settings\Luca\My Documents\config.txt")

Almost done, in the main part of the program, we gather the data, create the content of the email and send it out:

do
    let file = "spreads.csv"
    let spreads = processFile file DateTime.Today
    let mutable builder = new System.Text.StringBuilder()
    builder <- builder.AppendLine("Name\t\tCurrent\t\tStop")
    for s in spreads do
        builder <- builder.AppendLine(s.ToString())
    let password = getPassword()
    sendEmail "smtp.gmail.com" 587 "***@***.com" "***@***.com" "Alert Trigger Spread" (builder.ToString())
                                                                                           "lucabolg@gmail.com" password;;

Next stop, the WPF veneer on top of the file.

Downloading stock prices, divs and splits in F# updated to Visual Studio 2010

Fri, 26 Feb 2010 23:50:44 GMT

I’m working on a program to keep track of paired trades with trailing stops. I need to download stock prices, so I thought I might reuse some old code of mine. Here is the updated framework.

LChart: displaying charts in F# – Part III

Fri, 19 Feb 2010 18:03:00 GMT

The last post is here. In this post we’ll look at how things work under the cover and to why I came to believe that they shouldn’t work this way.

First of all each one of the functions to create charts looks something like this:

static member bar (?y,?x, ?isValueShownAsLabel, ?markerSize, ?markerStyle, ?color, ?xname, ?yname, ?seriesName, ?title, ?drawingStyle) =
    let c = Create (SeriesChartType.Bar, x, y, isValueShownAsLabel, markerSize, markerStyle, color, xname, yname, seriesName, title)
    c.Series.[0].["DrawingStyle"] <- defaultArg drawingStyle (c.Series.[0].["DrawingStyle"])
    c

This returns an object of type lc (this is the type of ‘c’). But lc inherits from Chart which is the main class in the Microsoft Chart Controls.

type lc() =
    inherit Chart()

I should have said at the start that you need to reference such controls.

#r "System.Windows.Forms.DataVisualization.dll"
open System.Collections
open System.Drawing
open System.IO
open System.Windows.Forms
open System.Windows.Forms.DataVisualization.Charting
open System.Windows.Forms.DataVisualization.Charting.Utilities
open System

It is convenient that the return value of each function is a subtype of Chart. You can then go and customize this object as you like (i.e. changing graphical appearance) before calling display. Given the Chart inherits from Control you can code the display method as follows:

let display (c:lc) =
    let copy () =
        let stream = new MemoryStream()
        c.SaveImage(stream, Imaging.ImageFormat.Bmp)
        let bmp = new Bitmap(stream)
        Clipboard.SetDataObject(bmp)
    c.KeyDown.Add(fun e -> if e.Control = true && e.KeyCode = Keys.C then copy ())
    let pressToCopy = "(press CTRL+C to copy)"
    let name = if c.Titles.Count = 0 then sprintf "%s %s " "lc" pressToCopy else sprintf "%s %s " c.Titles.[0].Text  pressToCopy
    let f = new Form(Text = name, Size = new Size(800,600), TopMost = true)
    c.Dock <- DockStyle.Fill
    f.Controls.Add(c)
    f.Show()
    c

Apart from a bit of convolutions to implement a Copy function, this just put the Chart control on a newly created Form. The Create method called inside bar looks like the following.

static let Create (chartType, x, y, isValueShownAsLabel, markerSize, markerStyle, color, xname, yname, seriesName, title) =
    let c = new lc()
    let a = new ChartArea()
    let s = new Series()
    s.ChartType <- chartType
    c.ChartAreas.Add(a)
    c.Series.Add(s)
    match x, y with
        | Some(x), None     -> failwith "You cannot pass only x to a chart drawing function"
        | Some(x), Some(y)  -> s.Points.DataBindXY(x, [|y|])
        | None, Some(y)     -> s.Points.DataBindY([|y|])
        | None, None        -> ()
    s.IsValueShownAsLabel <- defaultArg isValueShownAsLabel s.IsValueShownAsLabel
    s.MarkerSize <- defaultArg markerSize s.MarkerSize
    s.MarkerStyle <- defaultArg markerStyle s.MarkerStyle
    s.Color <- defaultArg color s.Color
    a.AxisX.MajorGrid.Enabled <- false
    a.AxisY.MajorGrid.Enabled <- false
    match xname with
    | Some(xname) ->
        a.AxisX.Title <- xname
        a.AxisX.TitleFont <- axisFont
        a.AxisX.TitleForeColor <- axisColor
    | _ -> ()
    match yname with
    | Some(yname) ->
        a.AxisY.Title <- yname
        a.AxisY.TitleFont <- axisFont
        a.AxisY.TitleForeColor <- axisColor
    | _ -> ()
    match seriesName with
    | Some(seriesName) -> s.Name <- seriesName
    | _ -> ()
    match title with
    | Some(title) ->
        let t = c.Titles.Add(title: string)
        t.Font <- titleFont
        t.ForeColor <- titleColor
    | _ -> ()
    c

Pretty standard imperative code here. Creating a chart and assigning its properties. Read the documentation for the Chart Control to understand what I’m doing here. I’m not even sure I remember what I’m doing. Given that we have our own lc class (which is a type of Chart) we can then override the ‘+’ operator and ‘++’ operator to do what is needed.

static member (+) (c1:lc, c2:lc) =
    let c = copyChart(c1)
    c1.ChartAreas |> Seq.iter (fun a -> addAreaAndSeries c a c1.Series)
    let lastArea = c.ChartAreas |> Seq.nth ((c.ChartAreas |> Seq.length) - 1)
    c2.Series |> Seq.iter(fun s -> c.Series.Add(copySeries s c lastArea.Name))
    let l = c.Legends.Add("")
    l.Font <- legendFont
    c
static member (++) (c1:lc, c2:lc) =
    let c = copyChart(c1)
    c1.ChartAreas |> Seq.iter (fun a -> addAreaAndSeries c a c1.Series)
    let lastArea = c.ChartAreas |> Seq.nth ((c.ChartAreas |> Seq.length) - 1)
    addAreaAndSeries c c2.ChartAreas.[0] c2.Series
    let firstArea = c.ChartAreas |> Seq.nth ((c.ChartAreas |> Seq.length) - 1)
    c2.ChartAreas |> Seq.skip 1 |> Seq.iter (fun a -> addAreaAndSeries c a c2.Series)
    c

Apart from some other utility functions, this is how it all works. Why do I say that it is wrong? It is my opinion that the right way to do it would be to use ’+’, ‘++’ and all the lc.XXX functions to create an object model that is completely independent from the Microsoft Chart controls. The display method would then translate it to the appropriate displayable Chart. It would work like a compiler translating to IL and then a Jitter producing native code. This would:

Make possible to do more interesting compositions of graphs. Now I’m very constrained in what I can do by the fact that I’m working directly with Chart objects
Make possible to change the backend. Using something different than Microsoft Chart controls to draw the chart

Why I have not done it? I didn’t know that was the right design until I used the wrong one. Now that I know, I have no time to do it.

Luca Bolognese leaves Microsoft

Fri, 19 Feb 2010 08:58:02 GMT

FYI I imported all the posts from the old MSDN blog to this one. Hence this post doesn’t make sense anymore. I left it here for memory sake. BTW: I am also back in msft now (30/11/2018).

This is my last post on this blog. My new blog is here: http://www.lucabol.com

I accepted a role as Director for Credit Suisse in London. I’m excited by the opportunity to work in the financial industry, a long-standing desire of mine. I’m also excited to write more F# code and to be closer to Italy, where my extended family is.

The past ten years in Microsoft have been a wild ride. I’m proud to have been part of ObjectSpaces, Generics, LINQ and F# (and much more …). I’ve been lucky to be able to post on this blog and present at conferences about such innovative technologies. It has all been a lot of fun. I’m sure the next ten years will be as good.

I’ll see you guys on my new blog.

LChart: displaying charts in F# – Part II

Wed, 17 Feb 2010 20:24:33 GMT

In the previous post on my old blog I showed how to display simple charts with LChart. In this one we’ll talk about more complex charts. I wanted to define a little language for graphs for the sake of creating a more complex chart in a single line of code. Remember, the scenario here is: I got some data, I want to display it quickly in the fsi. The language has two operators: ‘+’ and ‘++’.

’+’ allows you to to superimpose things on a chart as in the following example.

lc.scatter(y) + lc.line(y) |> display

Notice how I can superimpose two graphs of a different type and display the result.

Notice on the upper right corner a bizarre (Series1, Series2) legend. What I wanted to do, but didn’t get around to do, was to allow a syntax like

lc.scatter(y) + lc.line(y) + lc.legend("Players") |> display

It should be relatively trivial to add it to the code. Also notice that the ‘y’ parameter in the second chart is not needed. Data flows freely from left to right. So you can write the equivalent code below. This is a feature that caused all sort of grief. With hindsight it’s not worth the hassle. That’s why you always need to write your code at least twice to get it right.

lc.scatter(y) + lc.line() |> display

Some other, more elaborate charts follows. Notice how data flows from left to right until I introduce a new variable (i.e. the two lc.boxplot instructions plot different boxplots)

lc.scatter(y, markerSize = 10) + lc.column() + lc.boxplot() + lc.line()  + lc.column(x) + lc.boxplot()|> display

Things would be better with names for axis, titles and such. I’m not including them for the sake of simplicity.

lc.scatter(y, markerSize = 10) + lc.column() + (lc.line(x)  + lc.column()) + lc.scatter(markerSize = 20) |> display

If you remember the previous post, we talked about boxplots and how you generally want to have more than one on a graph. You can do that with the ‘+’ operator and get this ugly chart. More work is needed here.

lc.boxplot(y = y, xname = "Players", yname = "Ratings", title = "Players' Ratings", color = Color.Blue, whiskerPercentile = 5, percentile = 30,
    showAverage = false, showMedian = false, showUnusualValues = true) +  lc.boxplot(y = x) |> display

’++’ allows you to create new charts below the chart you already have as in:

let h = [1.;2.5;3.1;4.;4.8;6.0;7.5;8.;9.1;15.]
let w = h |> List.map (fun h -> h * 1.2)
lc.line(h) + lc.column() ++ lc.line(w) ++ lc.bubble() |> display

Notice the left to right flowing of information here as well. In the next installment we’ll take a look at how things are implemented and why it’s all wrong.

LChart: displaying charts in F# – Part I

Wed, 17 Feb 2010 11:20:00 GMT

I want to use F# as a exploratory data analysis language (like R). But I don’t know how to get the same nice graphic capabilities. So I decided to create them. Here is a library to draw charts in F#. It steals ideas from this book and this R package. It is nothing more than a wrapper on top of the Microsoft Chart Controls to give it a more ‘exploratory’ one line calling syntax. It is also rough work in progress: I don’t wrap all the chart types and there are bugs in the ones I wrap. Also the architecture is all wrong (more on this in another post). But it’s a start and it kind of works. Attached the full code.

I will continue this series in my new blog at wordpress: http://lucabolognese.wordpress.com/. The reason I need a new blog will be explained in an upcoming post.

Part II is now here.

Ok, let’s start. How do I draw a chart?

let x = [1.;2.5;3.1;4.;4.8;6.0;7.5;8.;9.1;15.]
let y = [1.6;2.1;1.4;4.;2.3;1.9;2.4;1.4;5.;2.9]
lc.scatter(x, y) |> display

X and Y are just some make up data. lc is the name of a class (????) and scatter is a static method on it. scatter doesn’t display the chart, it just produces a an object that represents the chart. Display displays the chart. The reason for using the bizarre lc static class is that I want it to be short so that it is easy to type in the fsi.exe. At the same time it needs to support optional parameters (which are not supported on top level functions in F#).

You get a window with this chart on it. You can press CTRL+C to copy it (as I did to post it here).

You might want to customize the chart a bit by passing some of these famous optional parameters:

lc.scatter(x = x, y = y, markerSize = 10, markerStyle = MarkerStyle.Diamond,
    xname = "Players", yname = "Ratings", title = "Players' Ratings")  |> display

Or you might want to print different types of charts:

lc.line(y = y, markerSize = 10, markerStyle = MarkerStyle.Diamond, xname = "Players", yname = "Ratings", title = "Players' Ratings", isValueShownAsLabel = true,
    color = Color.Red) |> display

lc.spline(x = x, y = y, markerSize = 10, markerStyle = MarkerStyle.Diamond, xname = "Players", yname = "Ratings",
    title = "Players' Ratings", isValueShownAsLabel = true, color = Color.Red) |> display

lc.stepline(x = x, y = y, markerSize = 10, markerStyle = MarkerStyle.Diamond, xname = "Players", yname = "Ratings",
    title = "Players' Ratings", isValueShownAsLabel = true, color = Color.Red) |> display

lc.bar(y = y, xname = "Players", yname = "Ratings", title = "Players' Ratings", isValueShownAsLabel = true,
    drawingStyle = "Emboss") |> display

lc.column(y = y, xname = "Players", yname = "Ratings", title = "Players' Ratings",
    isValueShownAsLabel = true, drawingStyle = "Cylinder") |> display

lc.boxplot(y = y, xname = "Players", yname = "Ratings", title = "Players' Ratings", color = Color.Blue, whiskerPercentile = 5, percentile = 30,
    showAverage = false, showMedian = false, showUnusualValues = true) |> display

Ok, the last one is weird. You probably want more than one boxplot in a chart. I’ll show you how to do that in the next post.

The next post will be on how to have more than one series on the same chart and more than one chart in the same windows. Something like the below:

ChartPlotter.fsx

Excel Financial Functions upgraded to Visual Studio 2010 RC

Tue, 16 Feb 2010 14:55:00 GMT

Here is a link to the home page: http://code.msdn.microsoft.com/FinancialFunctions/Wiki/View.aspx?title=Home

New version of LAgent compatible with VS 2010 RC online

Mon, 15 Feb 2010 09:36:54 GMT

My crazy agent framework now works with RC: http://code.msdn.microsoft.com/LAgent

A simpler F# MailboxProcessor

Fri, 12 Feb 2010 15:29:46 GMT

I always forget the pattern to use to create a functioning MailboxProcessor in F#. I mean, which piece has to be async and how to structure the recursive loop. When I find myself in that kind of a situation situation, my instincts scream at me: “Wrap it and make it work how your mind expects it to work”. So here is a simplification of the paradigm.

Let’s see how some standard MailboxProcessor code looks like:

let counter0 =
    MailboxProcessor.Start(fun inbox ->
        let rec loop n =
            async {
                    let! msg = inbox.Receive()
                    return! loop(n+msg) }
        loop 0)

This keeps a running sum of the messages it receives. The only part that is really unique to this guy is “n + msg”. All the rest is infrastructure.

You’d probably prefer to write code like the following:

let counter1 = MailboxProcessor.SpawnAgent((fun msg n -> msg + n), 0)

Yep, just one line of code. But, is it possible? Let’s look at one way of doing it:

type AfterError<'state> =
| ContinueProcessing of 'state
| StopProcessing
| RestartProcessing
type MailboxProcessor<'a> with
    static member public SpawnAgent<'b>(messageHandler :'a->'b->'b,
                                        initialState : 'b, ?timeout:'b -> int,
                                        ?timeoutHandler:'b -> AfterError<'b>,
                                        ?errorHandler:
                                            Exception -> 'a option -> 'b -> AfterError<'b>)
                                        : MailboxProcessor<'a> =
        let timeout = defaultArg timeout (fun _ -> -1)
        let timeoutHandler = defaultArg timeoutHandler (fun state ->
                                                                  ContinueProcessing(state))
        let errorHandler = defaultArg errorHandler (fun _ _ state ->
                                                                  ContinueProcessing(state))
        MailboxProcessor.Start(fun inbox ->
            let rec loop(state) = async {
                let! msg = inbox.TryReceive(timeout(state))
                try
                    match msg with
                    | None      -> match timeoutHandler state with
                                    | ContinueProcessing(newState)    ->
                                                                   return! loop(newState)
                                    | StopProcessing        -> return ()
                                    | RestartProcessing     -> return! loop(initialState)
                    | Some(m)   -> return! loop(messageHandler m state)
                with
                | ex -> match errorHandler ex msg state with
                        | ContinueProcessing(newState)    -> return! loop(newState)
                        | StopProcessing        -> return ()
                        | RestartProcessing     -> return! loop(initialState)
                }
            loop(initialState))

The funny formatting is because I have to fit it in the small horizontal space of this blog. In any case, this is just a simple (?) wrapper for the MailboxProcessor pattern. The function takes two necessary parameters and two optional ones:

messageHandler: a function to execute when a message comes in, it takes the message and the current state as parameters and returns the new state.
initialState: the initial state for the MailboxProcessor
timeoutHandler: a function that is executed whenever a timeout occurs. It takes as a parameter the current state and returns one of ContinueProcessing(newState), StopProcessing or RestartProcessing
errorHandler: a function that gets call if an exception is generated inside the messageHandler function. It takes the exception, the message, the current state and returns ContinueProcessing(newState), StopProcessing or RestartProcessing

An example of how to use errorHandler to implement the CountingAgent in the Expert F# book follows:

type msg = Increment of int | Fetch of AsyncReplyChannel<int> | Stop
exception StopException
type CountingAgent() =
    let counter = MailboxProcessor.SpawnAgent((fun msg n ->
                    match msg with
                    | Increment m ->  n + m
                    | Stop -> raise(StopException)
                    | Fetch replyChannel ->
                        do replyChannel.Reply(n)
                        n
                  ), 0, errorHandler = (fun _ _ _ -> StopProcessing))
    member a.Increment(n) = counter.Post(Increment(n))
    member a.Stop() = counter.Post(Stop)
    member a.Fetch() = counter.PostAndReply(fun replyChannel -> Fetch(replyChannel))
let counter2 = CountingAgent()
counter2.Increment(1)
counter2.Fetch()
counter2.Increment(2)
counter2.Fetch()
counter2.Stop()

Sometimes your agent doesn’t need a state, it is purely stateless. Something as simple as the following:

let echo = MailboxProcessor<_>.SpawnWorker(fun msg -> printfn "%s" msg)

You can easily make that happen by using this toned down version of an agent, called worker:

static member public SpawnWorker(messageHandler,  ?timeout, ?timeoutHandler,?errorHandler) =
    let timeout = defaultArg timeout (fun () -> -1)
    let timeoutHandler = defaultArg timeoutHandler (fun _ -> ContinueProcessing(()))
    let errorHandler = defaultArg errorHandler (fun _ _ -> ContinueProcessing(()))
    MailboxProcessor.SpawnAgent((fun msg _ -> messageHandler msg; ()),
                                 (), timeout, timeoutHandler,
                                 (fun ex msg _ -> errorHandler ex msg))

Given that they are parallel, you might want to run a whole bunch of them at the same time. You might want something that looks like a worker, but that, under the cover, execute each messageHandler in parallel. Something like:

type msg1 = Message1 | Message2 of int | Message3 of string
let a = MailboxProcessor.SpawnParallelWorker(function
                | Message1 -> printfn "Message1";
                | Message2 n -> printfn "Message2 %i" n;
                | Message3 _ -> failwith "I failed"
                , 10
                , errorHandler = (fun ex _ -> printfn "%A" ex; ContinueProcessing()))
a.Post(Message1)
a.Post(Message2(100))
a.Post(Message3("abc"))
a.Post(Message2(100))

In this example, the different messages, are likely to cause things to print out of order. Notice the number 10 above which is how many workers you want to process your messages. This is implemented by round-robin messages to the various workers:

static member public SpawnParallelWorker(messageHandler, howMany, ?timeout,
                                                        ?timeoutHandler,?errorHandler) =
    let timeout = defaultArg timeout (fun () -> -1)
    let timeoutHandler = defaultArg timeoutHandler (fun _ -> ContinueProcessing(()))
    let errorHandler = defaultArg errorHandler (fun _ _ -> ContinueProcessing(()))
    MailboxProcessor<'a>.SpawnAgent((fun msg (workers:MailboxProcessor<'a> array, index) ->
                                        workers.[index].Post msg
                                        (workers, (index + 1) % howMany))
                                    , (Array.init howMany
                                      (fun _ -> MailboxProcessor<'a>.SpawnWorker(
                                                 messageHandler, timeout, timeoutHandler,
                                                 errorHandler)), 0))

One drawback with the current code is that it doesn’t supports cancellations. It should be possible to wrap that too, but I haven’t done it (yet). If you don’t want to cut and paste the code, it is inside the AgentSystem.fs file here.

The F#unctional Londoners Meetup Group

Thu, 11 Feb 2010 09:24:05 GMT

If you live in London and work with F#, you should join http://www.meetup.com/FSharpLondon/

Also, if you don’t work with F#, but you’d like too, and you live in London you should join it. Also if you don’t live in London, but you’d like to and you’d like not to work on F# … Ok, you get the point.

LAgent: an agent framework in F# – Part X – ActiveObject

Thu, 10 Dec 2009 10:09:14 GMT

Download framework here.

All posts are here:

If you stare long enough at agents, you start to realize that they are just ‘glorified locks’. They are a convenient programming model to protect a resource from concurrent access. The programming model is convenient because both the client and the server can write their code without worrying about concurrency problems, and yet the program runs in parallel. Protecting a resource sounds a lot like state encapsulation and the concept of state encapsulation is what object orientation is all about.

So you start thinking if there is a way to enhance vanilla objects to make them agents. You want to reuse all the concepts that you are familiar with (i.e. inheritance, visibility rules, etc…) and you want your clients to call agents as if they were calling normal objects. Obviously, under the cover, the method calls won’t execute immediately, but they would be queued. Let’s look at an example.

This is our simple counter agent:

type CounterMessage =
| Add of int
| Print
let counterF = fun msg count ->
    match msg with
    | Add(i)    -> count + i
    | Print     -> printfn "The value is %i" count; count
let c1 = spawnAgent counterF 0
c1 <-- Add(3)
c1 <-- Print

As nice as this looks, there are unfamiliar things in this model:

The communication is through messages. This requires packing and unpacking which, albeit easy in F#, is unfamiliar and feels like machinery that we’d like to get rid off.
The management of state is bizarre, it gets passed into the lambda and returned from it instead of being represented as fields and properties on the agent

My best attempt at creating an object-like syntax follows:

type Counter() =
    let w = new WorkQueue()
    let mutable count = 0
    member c.Add x = w.Queue (fun () ->
        count <- count + x
        )
    member c.Print () = w.Queue (fun () ->
        printfn "The value is %i" count
        )

let c = new Counter()
c.Add 3
c.Print

With this syntax, you write your agents like you write your vanilla classes except:

You need a private field of type WorkQueue
You need to write your methods as lambdas passed to the WorkQueue.Queue function
Your methods cannot return values

The most worrisome of these constraints is 2. because you can easily forget about it. If you do forget, then everything compiles just fine, but it doesn’t do what you expect. That’s pure badness. I haven’t found a way to enforce it. This is a place where the language could help me. Other than that, the whole model works rather nicely.

Regarding the third point, you can concoct a programming model that allows you to return values from your methods. Here it is:

member c.CountTask = w.QueueWithTask(fun () ->
    count
    )
member c.CountAsync = w.QueueWithAsync(fun () ->
    count
    )

printfn "The count using Task is %i" (c.CountTask.Result)

The first method returns a Task; the second method returns an AsyncResultCell. Both are ways to represent a promise. The latter allows a natural integration with the async block in F# as in the following code:

Async.RunSynchronously (
            async {
                let! count = c.CountAsync
                printfn "The countusing Async is %i" count
            })

As for myself, I don’t like methods returning values. Every time I use them, I end up going back and thinking about my problem in a traditional way, aka as method calls that return results, instead of thinking about it in a more actor oriented fashion. I end up waiting for these promises to be materialized and, by doing so, I limit the amount of parallelism that I unleash. As a matter of fact, the whole business of hiding the message passing nature of the programming model is dubious. It makes for a nicer syntax, but you need to make an extra effort in your mind to translate it to what it really is: just message passing with a nice syntactical veneer. I haven’t decided yet which model I like the most.

You should have a sense of what WorkQueue is. In essence, it is a Mailbox of lambdas (look at the red bold code below).

type WorkQueue() =
    let workQueue = spawnWorker (fun f -> f())
    member w.Queue (f) = workQueue <-- f
    member w.QueueWithTask f : Task<'T> =
        let source = new TaskCompletionSource<_>()
        workQueue <-- (fun () -> f() |> source.SetResult)
        source.Task
    member w.QueueWithAsync (f:unit -> 'T) : Async<'T> =
        let result = new AsyncResultCell<'T>()
        workQueue <-- (fun () -> f() |> result.RegisterResult )
        result.AsyncWaitResult
    member w.Restart () = workQueue <-! Restart
    member w.Stop () = workQueue <-! Stop
    member w.SetErrorHandler(h) =
        let managerF = fun (_, name:string, ex:Exception, _, _, _) -> h name ex
        let manager = spawnWorker managerF
        workQueue <-! SetManager manager
    member w.SetName(name) = workQueue <-! SetName(name)
    member w.SetQueueHandler(g) = workQueue <-! SetWorkerHandler(g)
    member w.SetTimeoutHandler(timeout, f) = workQueue <-! SetTimeoutHandler(timeout, f)

I implemented all the services that are in the message passing model. The two are equivalent as expressing power goes. In case you wonder how a real piece of code looks like using this model, here is an ActiveObject version of the map reduce algorithm. One of these days, I will gather the strength to go trough this code and explain what it does, but not today 🙂

```fsharp

load “AgentSystem.fs”

open AgentSystem.LAgent open System open System.Collections open System.Collections.Generic open System.Threading type IOutput<’out_key, ‘out_value> = abstract Reduced: ‘out_key -> seq<’out_value> -> unit abstract MapReduceDone: unit -> unit type Mapper<’in_key, ‘in_value, ‘my_out_key, ‘out_value when ‘my_out_key : comparison> (map:’in_key -> ‘in_value -> seq<’my_out_key * ‘out_value>, i, partitionF) = let w = new WorkQueue() let mutable reducerTracker: BitArray = null let mutable controller = Unchecked.defaultof<Controller<’in_key, ‘in_value, ‘my_out_key, ‘out_value>> let mutable reducers = Unchecked.defaultof<Reducer<’in_key, ‘in_value, ‘my_out_key, ‘out_value> array> member m.Init c reds = w.Queue (fun () -> controller <- c reducers <- reds reducerTracker <- new BitArray(reducers.Length, false)) member m.Process inKey inValue = w.Queue (fun () -> let outKeyValues = map inKey inValue outKeyValues |> Seq.iter (fun (outKey, outValue) -> let reducerUsed = partitionF outKey (reducers.Length) reducerTracker.Set(reducerUsed, true) reducers.[reducerUsed].Add(outKey, outValue))) member m.Done () = w.Queue (fun () -> controller.MapDone i reducerTracker) member m.Stop () = w.Stop () and Reducer<’in_key, ‘in_value, ‘out_key, ‘out_value when ‘out_key : comparison>(reduce:’out_key -> seq<’out_value> -> seq<’out_value>, i, output:IOutput<’out_key, ‘out_value>) = let w = new WorkQueue() let mutable workItems = new List<’out_key * ‘out_value>() let mutable controller = Unchecked.defaultof<Controller<’in_key, ‘in_value, ‘out_key, ‘out_value>> member r.Init c = w.Queue (fun () -> controller <- c) member r.StartReduction () = w.Queue (fun () -> workItems |> Seq.groupBy fst |> Seq.sortBy fst |> Seq.map (fun (key, values) -> (key, reduce key (values |> Seq.map snd))) |> Seq.iter (fun (key, value) -> output.Reduced key value) controller.ReductionDone i) member r.Add (outKey:’out_key, outValue:’out_value) : unit = w.Queue (fun () -> workItems.Add((outKey, outValue))) member m.Stop () = w.Stop () and Controller<’in_key, ‘in_value, ‘out_key, ‘out_value when ‘out_key : comparison>(output:IOutput<’out_key, ‘out_value>) = let w = new WorkQueue() let mutable mapperTracker: BitArray = null let mutable reducerUsedByMappers: BitArray = null let mutable reducerDone: BitArray = null let mutable mappers = Unchecked.defaultof<Mapper<’in_key, ‘in_value, ‘out_key, ‘out_value> array> let mutable reducers = Unchecked.defaultof<Reducer<’in_key, ‘in_value, ‘out_key, ‘out_value> array> let BAtoSeq (b:BitArray) = [for x in b do yield x] member c.Init maps reds = w.Queue (fun () -> mappers <- maps reducers <- reds mapperTracker <- new BitArray(mappers.Length, false) reducerUsedByMappers <- new BitArray(reducers.Length, false) reducerDone <- new BitArray(reducers.Length, false)) member c.MapDone (i : int) (reducerTracker : BitArray) : unit = w.Queue (fun () -> mapperTracker.Set(i, true) let reducerUsedByMappers = reducerUsedByMappers.Or(reducerTracker) if not( BAtoSeq mapperTracker |> Seq.exists(fun bit -> bit = false)) then BAtoSeq reducerUsedByMappers |> Seq.iteri (fun i r -> if r = true then reducers.[i].StartReduction ()) mappers |> Seq.iter (fun m -> m.Stop ()) ) member c.ReductionDone (i: int) : unit = w.Queue (fun () -> reducerDone.Set(i, true) if BAtoSeq reducerDone |> Seq.forall2 (fun x y -> x = y) (BAtoSeq reducerUsedByMappers) then output.MapReduceDone () reducers |> Seq.iter (fun r -> r.Stop ()) c.Stop() ) member m.Stop () = w.Stop () let mapReduce (inputs:seq<’in_key * ‘in_value>) (map:’in_key -> ‘in_value -> seq<’out_key * ‘out_value>) (reduce:’out_key -> seq<’out_value> -> seq<’out_value>) (output:IOutput<’out_key, ‘out_value>) M R partitionF = let len = inputs |> Seq.length let M = if len < M then len else M let mappers = Array.init M (fun i -> new Mapper<’in_key, ‘in_value, ‘out_key, ‘out_value>(map, i, partitionF)) let reducers = Array.init R (fun i -> new Reducer<’in_key, ‘in_value, ‘out_key, ‘out_value>(reduce, i, output)) let controller = new Controller<’in_key, ‘in_value, ‘out_key, ‘out_value>(output) mappers |> Array.iter (fun m -> m.Init controller reducers) reducers |> Array.iter (fun r -> r. Init controller ) controller.Init mappers reducers inputs |> Seq.iteri (fun i (inKey, inValue) -> mappers.[i % M].Process inKey inValue) mappers |> Seq.iter (fun m -> m.Done ()) let partitionF = fun key M -> abs(key.GetHashCode()) % M let map = fun (fileName:string) (fileContent:string) -> let l = new List() let wordDelims = [|’ ‘;’,’;’;’;’.’;’:’;’?’;’!’;’(‘;’)’;’n’;’t’;’f’;’r’;’b’|] fileContent.Split(wordDelims) |> Seq.iter (fun word -> l.Add((word, 1))) l :> seq let reduce = fun key (values:seq) -> [values |> Seq.sum] |> seq let printer () = { new IOutput with member o.Reduced key values = printfn “%A %A” key values member o.MapReduceDone () = printfn “All done!!”} let testInput = [“File1”, “I was going to the airport when I saw someone crossing”; “File2”, “I was going home when I saw you coming toward me”] mapReduce testInput map reduce (printer ()) 2 2 partitionF open System.IO open System.Text let gatherer(step) = let w = new WorkQueue() let data = new List() let counter = ref 0 { new IOutput with member o.Reduced key values = w.Queue (fun () -> if !counter % step = 0 then printfn “Processed %i words. Now processing %s” !counter key data.Add((key, values |> Seq.hd)) counter := !counter + 1) member o.MapReduceDone () = w.Queue (fun () -> data |> Seq.distinctBy (fun (key, ) -> key.ToLower()) |> Seq.filter (fun (key, ) -> not(key = “” || key = “”” || (fst (Double.TryParse(key))))) |> Seq.to_array |> Array.sortBy snd |> Array.rev |> Seq.take 20 |> Seq.iter (fun (key, value) -> printfn “%A\t%A” key value) printfn “All done!!”) } let splitBook howManyBlocks fileName = let buffers = Array.init howManyBlocks (fun _ -> new StringBuilder()) fileName |> File.ReadAllLines |> Array.iteri (fun i line -> buffers.[i % (howManyBlocks)].Append(line) |> ignore) buffers let blocks1 = SOURCE_DIRECTORY + “kjv10.txt” |> splitBook 100 let blocks2 = SOURCE_DIRECTORY + “warandpeace.txt” |> splitBook 100 let input = blocks1 |> Array.append blocks2 |> Array.mapi (fun i b -> i.ToString(), b.ToString()) //mapReduce input map reduce (gatherer(1000)) 20 20 partitionF type BookSplitter () = let blocks = new List() member b.Split howManyBlocks fileName = let b = fileName |> splitBook howManyBlocks |> Array.mapi (fun i b -> i.ToString(), b.ToString()) blocks.AddRange(b) member b.Blocks () = blocks.ToArray() :> seq type WordCounter () = let w = new WorkQueue() let words = new Dictionary() let worker(wordCounter:WordCounter, ev:EventWaitHandle) = let w1 = new WorkQueue() { new IOutput with member o.Reduced key values = w1.Queue (fun() -> wordCounter.AddWord key (values |> Seq.hd)) member o.MapReduceDone () = w1.Queue(fun () -> ev.Set() |> ignore) } member c.AddWord word count = let exist, value = words.TryGetValue(word) if exist then words.[word] <- value + count else words.Add(word, count) member c.Add fileName = w.Queue (fun () -> let s = new BookSplitter() fileName |> s.Split 100 let ev = new EventWaitHandle(false, EventResetMode.AutoReset) let blocks = s.Blocks () mapReduce blocks map reduce (worker(c, ev)) 20 20 partitionF ev.WaitOne() |> ignore ) member c.Words = w.QueueWithAsync (fun () -> words |> Seq.to_array |> Array.map (fun kv -> kv.Key, kv.Value) ) let wc = new WordCounter() wc.Add (SOURCE_DIRECTORY + “kjv10.txt”) wc.Add (SOURCE_DIRECTORY + “warandpeace.txt”) let wordsToPrint = async { let! words = wc.Words return words |> Seq.distinctBy (fun (key, ) -> key.ToLower()) |> Seq.filter (fun (key, ) -> not(key = “” || key = “”” || (fst (Double.TryParse(key))))) |> Seq.to_array |> Array.sortBy snd |> Array.rev |> Seq.take 20 |> Seq.iter (fun (key, value) -> printfn “%A\t%A” key value)} Async.RunSynchronously wordsToPrint Thread.Sleep(15000) printfn “Closed session”

My PDC session is online - “Future directions for C# and Visual Basic”

Mon, 23 Nov 2009 13:17:44 GMT

In case you are training to play the part of a first generation Italian immigrant in a Broadway show or if you want to understand why I never short msft stock: http://microsoftpdc.com/Sessions/FT11

Luca at PDC 2009 next week

Fri, 13 Nov 2009 10:49:18 GMT

I’ll be in Los Angeles next week for PDC 2009. My session is called “Future Directions for C# and Visual Basic” and it comes on Tuesday immediately after the first keynote.

I’m planning on spending the first half of the session talking about the biggest trends influencing C# and VB. The second part will be about more future looking features we are playing with these days.

My deck has just two slides, as always. It’s all about watching me typing code on stage, hoping that I make crazy errors …

There are several other interesting language related sessions at PDC. Here is a list of them (please notice that the times might change). Also feel free to drop by the languages booth and chat. My team and I will be there most of the time.

Day	Time	Title	Presenter	Room
Tues 11/17	11:00 - 12:00	Future Directions for C# and Visual Basic	Luca Bolognese	Hall F
Tues 11/17	11:00 - 12:00	Accelerated Windows Application Development with Microsoft Visual C++ 2010	Boris Jabes	408B
Tues 11/17	12:30 - 13:15	Dynamic Binding in C# 4	Mads Torgersen	Hall F
Tues 11/17	12:30 - 13:15	Using Dynamic Languages to Build Scriptable Applications	Dino Viehland	403AB
Tues 11/17	13:30 - 14:30	C++ Forever: Interactive Applications in the Age of Manycore	Rick Molloy	515B
Tues 11/17	16:30 - 17:30	Manycore and the Microsoft .NET Framework 4: A Match Made in Microsoft Visual Studio 2010	Stephen Toub	502A
Tues 11/17	16:30 - 17:30	Code Contracts and Pex: Power Charge Your Assertions and Unit Tests	Mike Barnett, Nikolai Tillmann	408A
Wed 11/18	11:30 - 12:30	Microsoft Perspectives on the Future of Programming	Panel	Petree Hall C
Wed 11/18	12:00 - 13:00	F# for Architects – Hitting the Sweet Spot	Chris Smith	VS and .NET Pavillion
Wed 11/18	13:00 - 13:45	Code Like the Wind with Microsoft Visual Basic 2010	Lucian Wischik	Petree Hall D
Wed 11/18	13:00 - 13:45	Future of Garbage Collection	Patrick Dussud	Petree Hall C
Wed 11/18	13:00 - 13:45	Microsoft Project Code Name “M”: The Data and Modeling Language	Don Box, Jeff Pinkston	408A
Thu 11/19	08:30 - 09:30	PLINQ: LINQ, but Faster!	Ed Essey, Igor Ostrovsky	515A
Thu 11/19	10:00 - 11:00	A Lap around Microsoft Visual Studio 2010 for the Visual Basic Developer	Lisa Feigenbaum	VS and .NET Pavillion
Thu 11/19	10:00 - 11:00	Axum: A .NET Language for Safe and Scalable Concurrency	Niklas Gustafsson	515A
Thu 11/1 9	11:30 - 12:30	F# for Parallel and Asynchronous Programming	Luke Hoban	515A
Thu 11/19	12:45 - 13:30	Microsoft Visual C# IDE Tips and Tricks	DJ Park	Petree Hall D
Thu 11/19	12:45 - 13:30	Microsoft Visual Basic IDE Tips and Tricks	Dustin Campbell	Petree Hall C

Becoming really rich with C#

Tue, 22 Sep 2009 19:40:00 GMT

Or maybe not, please do not hold me responsible if you lose money following this system. Having said that, it is my opinion that there are very few concepts that are important in investing. Three big ones are value, diversification and momentum. This post is about the latter two and how to use C# to create a simple trading system that uses both.

Diversification is ‘not put all your eggs in one basket’ (contrary to ‘put all of them in one basket and watch that basket’). I don’t believe you can ‘watch’ very much in financial markets, so I tend to prefer diversification.

Momentum is a mysterious tendency of financial prices that have risen the most in the recent past, to continue outperforming in the close future. In essence, buying the top stocks/sectors/asset classes tends to outperform buying the bottom ones over horizons from three months to one year.

The idea then is to rank some assets (i.e. ETFs) by how fast they have risen in the past, go long the top ones and short the bottom ones. There are hundreds of variations of this basic strategy, we’ll add the rule that we won’t buy assets that are below their 200 days moving average or sell short assets that are above it.

I’m writing this code with VS 2010 Beta 2 (which hasn’t shipped yet). It should be trivial to modify it to run on B1 (or maybe it does run on it already). I attach the code and data files to this post.

struct Event {
    internal Event(DateTime date, double price) { Date = date; Price = price; }
    internal readonly DateTime Date;
    internal readonly double Price;
}

We’ll use this simple structure to load the closing price for a particular date. My use of internal is kind of bizarre. Actually the whole code might look strange. It is an interesting (maybe un-elegant) mix of object orientation and functional programming.

class Summary {
    internal Summary(string ticker, string name, string assetClass,
                    string assetSubClass, double? weekly, double? fourWeeks,
                    double? threeMonths, double? sixMonths, double? oneYear,
                    double? stdDev, double price, double? mav200) {
        Ticker = ticker;
        Name = name;
        AssetClass = assetClass;
        AssetSubClass = assetSubClass;
        // Abracadabra ...
        LRS = (fourWeeks + threeMonths + sixMonths + oneYear) / 4;
        Weekly = weekly;
        FourWeeks = fourWeeks;
        ThreeMonths = threeMonths;
        SixMonths = sixMonths;
        OneYear = oneYear;
        StdDev = stdDev;
        Mav200 = mav200;
        Price = price;
    }
    internal readonly string Ticker;
    internal readonly string Name;
    internal readonly string AssetClass;
    internal readonly string AssetSubClass;
    internal readonly double? LRS;
    internal readonly double? Weekly;
    internal readonly double? FourWeeks;
    internal readonly double? ThreeMonths;
    internal readonly double? SixMonths;
    internal readonly double? OneYear;
    internal readonly double? StdDev;
    internal readonly double? Mav200;
    internal double Price;
    internal static void Banner() {
        Console.Write("{0,-6}", "Ticker");
        Console.Write("{0,-50}", "Name");
        Console.Write("{0,-12}", "Asset Class");
        //Console.Write("{0,-30}t", "Asset SubClass";
        Console.Write("{0,4}", "RS");
        Console.Write("{0,4}", "1Wk");
        Console.Write("{0,4}", "4Wk");
        Console.Write("{0,4}", "3Ms");
        Console.Write("{0,4}", "6Ms");
        Console.Write("{0,4}", "1Yr");
        Console.Write("{0,6}", "Vol");
        Console.WriteLine("{0,2}", "Mv");
        //Console.Write("{0,6}", "Pr");
        //Console.WriteLine("{0,6}", "M200");
    }
    internal void Print() {
        Console.Write("{0,-6}", Ticker);
        Console.Write("{0,-50}", new String(Name.Take(48).ToArray()));
        Console.Write("{0,-12}", new String(AssetClass.Take(10).ToArray()));
        //Console.Write("{0,-30}t", new String(AssetSubClass.Take(28).ToArray()));
        Console.Write("{0,4:N0}", LRS * 100);
        Console.Write("{0,4:N0}", Weekly * 100);
        Console.Write("{0,4:N0}", FourWeeks * 100);
        Console.Write("{0,4:N0}", ThreeMonths * 100);
        Console.Write("{0,4:N0}", SixMonths * 100);
        Console.Write("{0,4:N0}", OneYear * 100);
        Console.Write("{0,6:N0}", StdDev * 100);
        if (Price <= Mav200)
            Console.WriteLine("{0,2}", "X");
        else
            Console.WriteLine();
        //Console.Write("{0,6:N2}", Price);
        //Console.WriteLine("{0,6:N2}", Mav200);
    }
}

The class Summary above is how I want to present my results. A few comments on the code. I use Nullable because some of this values can be null (i.e. not enough history), but I still don’t want to worry about it. It ends up working rather neatly.

I also print the results out to Console, which is crazy. I really should be using WPF/Silverlight as the presentation layer. Also the {0,4:N0} notation might be unfamiliar to some of you, but this is how mad Console guys like myself avoid using real UI frameworks. Sometimes we print things in color too.

The real meat is in the following line:

LRS = (fourWeeks + threeMonths + sixMonths + oneYear) / 4;

That is our highway to richness. It’s a very elaborated quant formula, never before shown, that calculate a magick relative strength (aka momentum) factor as the average of the performance of four weeks, three months, six months and one year.

class TimeSeries {
    internal readonly string Ticker;
    readonly DateTime _start;
    readonly Dictionary<DateTime, double> _adjDictionary;
    readonly string _name;
    readonly string _assetClass;
    readonly string _assetSubClass;
    internal TimeSeries(string ticker, string name, string assetClass, string assetSubClass, 
                                                                IEnumerable<Event> events) {
        Ticker = ticker;
        _name = name;
        _assetClass = assetClass;
        _assetSubClass = assetSubClass;
        _start = events.Last().Date;
        _adjDictionary = events.ToDictionary(e => e.Date, e => e.Price);
    }

I then built myself a little TimeSeries class that represents a series of (date, price). I choose a dictionary to store it because of my assumption that I will be accessing it by date a lot. In retrospect, I was kind of right and kind of wrong. It doesn’t really matter much.

bool GetPrice(DateTime when, out double price, out double shift) {
    // To nullify the effect of hours/min/sec/millisec being different from 0
    when = new DateTime(when.Year, when.Month, when.Day);
    var found = false;
    shift = 1;
    double aPrice = 0;
    while (when >= _start && !found) {
        if (_adjDictionary.TryGetValue(when, out aPrice)) {
            found = true;
        }
        when = when.AddDays(-1);
        shift -= 1;
    }
    price = aPrice;
    return found;
}

A TimeSeries can give you back the price at a particular date. This looks bizarre and complex, but there is a reason for it. I might ask for a date that doesn’t have a price associated with it (i.e. holidays, week-ends). In such cases I want to return the previous price which could be N days in the past.

I also want to return how many days in the past I had to go, so that other calculations (i.e. Return) can modify their end date by the same amount. Also I might not find such a price at all, in which case I don’t want to throw an exception, but instead notify the caller. In retrospect, I should have used double? to signify ‘price not found’.

double? GetReturn(DateTime start, DateTime end) {
    var startPrice = 0.0;
    var endPrice = 0.0;
    var shift = 0.0;
    var foundEnd = GetPrice(end, out endPrice, out shift);
    var foundStart = GetPrice(start.AddDays(shift), out startPrice, out shift);
    if (!foundStart || !foundEnd)
        return null;
    else
        return endPrice / startPrice - 1;
}

We can now go and calculate the return between two dates. Also the TimeSeries object needs to perform a little more calculations.

internal double? LastWeekReturn() {
        return GetReturn(DateTime.Now.AddDays(-7), DateTime.Now);
    }
    internal double? Last4WeeksReturn() {
        return GetReturn(DateTime.Now.AddDays(-28), DateTime.Now);
    }
    internal double? Last3MonthsReturn() {
        return GetReturn(DateTime.Now.AddMonths(-3), DateTime.Now);
    }
    internal double? Last6MonthsReturn() {
        return GetReturn(DateTime.Now.AddMonths(-6), DateTime.Now);
    }
    internal double? LastYearReturn() {
        return GetReturn(DateTime.Now.AddYears(-1), DateTime.Now);
    }
    internal double? StdDev() {
        var now = DateTime.Now;
        now = new DateTime(now.Year, now.Month, now.Day);
        var limit = now.AddYears(-3);
        var rets = new List<double>();
        while (now >= _start.AddDays(12) && now >= limit) {
            var ret = GetReturn(now.AddDays(-7), now);
            rets.Add(ret.Value);
            now = now.AddDays(-7);
        }
        var mean = rets.Average();
        var variance = rets.Select(r => Math.Pow(r - mean, 2)).Sum();
        var weeklyStdDev = Math.Sqrt(variance / rets.Count);
        return weeklyStdDev * Math.Sqrt(40);
    }
    internal double? MAV200() {
        return _adjDictionary
               .ToList()
               .OrderByDescending(k => k.Key)
               .Take(200)
               .Average(k => k.Value);
    }
    internal double TodayPrice() {
        var price = 0.0;
        var shift = 0.0;
        GetPrice(DateTime.Now, out price, out shift);
        return price;
    }
    internal Summary GetSummary() {
        return new Summary(Ticker, _name, _assetClass, _assetSubClass, 
                           LastWeekReturn(), Last4WeeksReturn(), Last3MonthsReturn(),
                           Last6MonthsReturn(), LastYearReturn(), StdDev(), TodayPrice(),
                           MAV200());
    }

Nothing particularly interesting in this code. Just a bunch of calculations. The MAV200 is the 200 days moving average of closing prices. It shows a more functional way of doing things. The StdDev function is instead very imperative.

We now can work on downloading the prices. This is how you construct the right URL:

static string CreateUrl(string ticker, DateTime start, DateTime end) {
    return @"http://ichart.finance.yahoo.com/table.csv?s=" + ticker + "&a="
            + (start.Month - 1).ToString() + "&b=" + start.Day.ToString() + "&c="
            + start.Year.ToString() + "&d=" + (end.Month - 1).ToString() + "&e="
            + end.Day.ToString() + "&f=" + end.Year.ToString() + "&g=d&ignore=.csv";
}

And let’s set how many concurrent connections we are going to use …

ServicePointManager.DefaultConnectionLimit = 10;

On my machine, setting this number too high causes errors to be returned. I’m not sure on which side of the connection the problem lies.

We can then load all the tickers we want to load from a file. One of the files has Leveraged ETFs, which I want to filter out because they tend to pop up always at the top.

var tickers =
    //File.ReadAllLines("ETFs.csv")
    //File.ReadAllLines("ETFTest.csv")
    File.ReadAllLines("AssetClasses.csv")
    .Skip(1)
    .Select(l => l.Split(new[] { ',' }))
    .Where(v => v[2] != "Leveraged")
    .Select(values => Tuple.Create(values[0], values[1], values[2], values[3]))
    .ToArray();
var len = tickers.Length;
var start = DateTime.Now.AddYears(-2);
var end = DateTime.Now;
var cevent = new CountdownEvent(len);
var summaries = new Summary[len];

And then load all of them, making sure to make an asynchronous call so not to keep the thread busy.

for(var i = 0; i < len; i++)  {
    var t = tickers[i];
    var url = CreateUrl(t.Item1, start, end);
    using (var webClient = new WebClient()) {
        webClient.DownloadStringCompleted +=
                          new DownloadStringCompletedEventHandler(downloadStringCompleted);
        webClient.DownloadStringAsync(new Uri(url), Tuple.Create(t, cevent, summaries, i));
    }
}
cevent.Wait();

Notice the use of a Countdown event to wait for all the thread to complete before printing out the results. Also notice the new Tuple class used to package things to send around.

We can then print out the top and bottom 15%:

var top15perc =
        summaries
        .Where(s => s.LRS.HasValue)
        .OrderByDescending(s => s.LRS)
        .Take((int)(len * 0.15));
var bottom15perc =
        summaries
        .Where(s => s.LRS.HasValue)
        .OrderBy(s => s.LRS)
        .Take((int)(len * 0.15));
Console.WriteLine();
Summary.Banner();
Console.WriteLine("TOP 15%");
foreach(var s in top15perc)
    s.Print();
Console.WriteLine();
Console.WriteLine("Bottom 15%");
foreach (var s in bottom15perc)
    s.Print();

Here is what we do when a request comes back with data:

static void downloadStringCompleted(object sender, DownloadStringCompletedEventArgs e) {
    var bigTuple =
             (Tuple<Tuple<string, string, string, string>, CountdownEvent, Summary[], int>)
              e.UserState;
    var tuple = bigTuple.Item1;
    var cevent = bigTuple.Item2;
    var summaries = bigTuple.Item3;
    var i = bigTuple.Item4;
    var ticker = tuple.Item1;
    var name = tuple.Item2;
    var asset = tuple.Item3;
    var subAsset = tuple.Item4;
    if (e.Error == null) {
        var adjustedPrices =
                e.Result
                .Split(new[] { 'n' })
                .Skip(1)
                .Select(l => l.Split(new[] { ',' }))
                .Where(l => l.Length == 7)
                .Select(v => new Event(DateTime.Parse(v[0]), Double.Parse(v[6])));
        var timeSeries = new TimeSeries(ticker, name, asset, subAsset, adjustedPrices);
        summaries[i] = timeSeries.GetSummary();
        cevent.Signal();
        Console.Write("{0} ", ticker);
    }
    else {
        Console.WriteLine("[{0} ERROR] ", ticker);
        //Console.WriteLine(e.Error);
        summaries[i] = new Summary(ticker, name, "ERROR", "ERROR", 0, 0, 0, 0, 0, 0,0,0);
        cevent.Signal();
    }
}

We first unpack the Tuple we sent out originally, we then extract the Date and Price, create a Summary object and store it in the summaries array. It’s important to remember to Signal to the cevent in the error case as well because we want to print out the results even if some downloading failed.

And here is what you get for your effort:

SystemCodeAndData.zip

LAgent: an agent framework in F# – Part IX – Counting words …

Fri, 18 Sep 2009 14:43:00 GMT

Download framework here.

All posts are here:

Let’s now use our mapReduce to do something more interesting, for example finding the frequency of words in several books. Now the agent that processes the output needs to be a bit more complex.

let gathererF = fun msg (data:List<string * int>, counter, step) ->
                    match msg with
                    | Reduced(key, value)   ->
                        if counter % step = 0 then
                            printfn "Processed %i words. Now processing %s" counter key
                        data.Add((key, value |> Seq.hd))
                        data, counter + 1, step
                    | MapReduceDone         ->
                        data
                        |> Seq.distinctBy (fun (key, _) -> key.ToLower())
                        |> Seq.filter (fun (key, _) -> not(key = "" || key = """ ||
                                                             (fst (Double.TryParse(key)))))
                        |> Seq.to_array
                        |> Array.sortBy snd
                        |> Array.rev
                        |> Seq.take 20
                        |> Seq.iter (fun (key, value) -> printfn "%A\t%A" key value)
                        printfn "All done!!"
                        data, counter, step
let gatherer = spawnAgent gathererF (new List<string * int>(), 0, 1000)

Every time a new word is reduced, a message is printed out and the result is added to a running list. When everything is done such a list is printed out by first manipulating it to reduce weirdness and limit the number of items. BTW: there are at least two bugs in this code, maybe more (late night quick-and-dirty-see-if-the-algo-works kind of coding).

We want to maximize the number of processors to use, so let’s split the books in chunks so that they can be operated in parallel. The code below roughly does it (I say roughly because it doesn’t chunk the lines in the right order, but for this particular case it doesn’t matter).

let gatherer = spawnAgent gathererF (new List<string * int>(), 0, 1000)
let splitBook howManyBlocks fileName =
    let buffers = Array.init howManyBlocks (fun _ -> new StringBuilder())
    fileName
    |> File.ReadAllLines
    |> Array.iteri (fun i line -> buffers.[i % (howManyBlocks)].Append(line) |> ignore)
    buffers
let blocks1 = "C:UserslucabolDesktopAgentsAgentskjv10.txt" |> splitBook 100
let blocks2 = "C:UserslucabolDesktopAgentsAgentswarandpeace.txt" |> splitBook 100
let input =
    blocks1
    |> Array.append blocks2
    |> Array.mapi (fun i b -> i.ToString(), b.ToString())

And let’s execute!!

mapReduce input map reduce gatherer 20 20 partitionF

On my machine I get the following, which could be the right result.

```fsharp “a” 16147 “And” 13071 “I” 11349 “unto” 8125 “as” 6400 “her” 5865 “which” 5544 “from” 5378 “at” 5175 “on” 5155 “have” 5135 “me” 5068 “my” 4629 “this” 3782 “out” 3653 “ye” 3399 “when” 3312 “an” 2841 “upon” 2558 “so” 2489 All done!!

LAgent: an agent framework in F# – Part VIII - Implementing MapReduce (user model)

Fri, 04 Sep 2009 13:57:48 GMT

Download framework here.

All posts are here:

For this post I use a newer version of the framework that I just uploaded on CodeGallery. In the process of using LAgent I grew more and more unhappy with the weakly typed way of sending messages. The code that implements that feature is nasty: full of upcasts and downcasts. I was losing faith in it. Bugs were cropping up in all sorts of scenarios (i.e. using generic union types as messages).

In the end I decided to re-architecture the framework so to make it strongly typed. In essence now each agent can just receive messages of a single type. The limitations that this design choice introduces (i.e. more limited hot swapping) are compensated by the catching of errors at compile time and the streamlining of the code. I left the old framework on the site in case you disagree with me.

In any case, today’s post is about MapReduce. It assumes that you know what it is (link to the original Google paper that served as inspiration is here: Google Research Publication- MapReduce). What would it take to implement an in-memory MapReduce using my agent framework?

Let’s start with the user model.

let mapReduce   (inputs:seq<'in_key * 'in_value>)
                (map:'in_key -> 'in_value -> seq<'out_key * 'out_value>)
                (reduce:'out_key -> seq<'out_value> -> seq<'reducedValues>)
                outputAgent
                M R partitionF =

mapReduce takes seven parameters:

inputs: a sequence of input key/value pairs.
map: this function operates on each input key/value pair. It returns a sequence of output key/value pairs. The type of the output sequence can be different from the type of the inputs.
reduce: this function operates on an output key and all the values associated with it. It returns a sequence of reduced values (i.e. the average of all the values for this key)
ouputAgent: this is the agent that gets notified every time a new output key has been reduced and at the end when all the operation ends.
M: how many mapper agents to instantiate
R: how many reducer agents to instantiate
partitionF: the partition function used to choose which of the reducers is associated with a key

Let’s look at how to use this function to find how often each word is used in a set of files. First a simple partition function can be defined as:

let partitionF = fun key M -> abs(key.GetHashCode()) % M

Given a key and some buckets, it picks one of the buckets. Its type is: ‘a –> int –> int, so it’s fairly reusable.

Let’s also create a basic agent that just prints out the reduced values:

let printer = spawnWorker (fun msg ->
                            match msg with
                            | Reduced(key, value)   -> printfn "%A %A" key value
                            | MapReduceDone         -> printfn "All done!!")

The agent gets notified whenever a new key is reduced or the algorithm ends. It is useful to be notified immediately instead of waiting for everything to be done. If I hadn’t written this code using agents I would have not realized that possibility. I would simply have framed the problem as a function that takes an input and returns an output. Agents force you to think explicitly about the parallelism in your app. That’s a good thing.

The mapping function simply split the content of a file into words and adds a word/1 pair to the list. I know that there are much better ways to do this (i.e. regular expressions for the parsing and summing words counts inside the function), but I wanted to test the basic framework capabilities and doing it this way does it better.

let map = fun (fileName:string) (fileContent:string) ->
            let l = new List<string * int>()
            let wordDelims = [|' ';',';';';'.';':';'?';'!';'(';')';'n';'t';'f';'r';'b'|]
            fileContent.Split(wordDelims) |> Seq.iter (fun word -> l.Add((word, 1)))
            l :> seq<string * int>

The reducer function simply sums the various word statistics sent by the mappers:

let reduce = fun key (values:seq<int>) -> [values |> Seq.sum] |> seq<int>

Now we can create some fake input to check that it works:

let testInput = ["File1", "I was going to the airport when I saw someone crossing";
                               "File2", "I was going home when I saw you coming toward me"]

And execute the mapReduce:

mapReduce testInput map reduce printer 2 2 partitionF

On my machine I get the following. You might get a different order because of the async/parallel processing involved. If I wanted a stable order I would need to change the printer agent to cache results on Reduced and process them on MapReduceDone (see next post).

“I” [4]

“crossing” [1]

“going” [2]

“home” [1]

“me” [1]

“the” [1]

“toward” [1]

“airport” [1]

“coming” [1]

“saw” [2]

“someone” [1]

“to” [1]

“was” [2]

“when” [2]

“you” [1]

In the next post we’ll process some real books …

LAgent: an agent framework in F# – part VII – An auction application

Fri, 10 Jul 2009 16:14:00 GMT

Download framework here.

All posts are here:

Here is an application that uses the framework we have been creating. It is an auction application and it is described in more detail here.

Let’s go through it.

type AuctionMessage =
  | Offer of int * AsyncAgent // Make a bid
  | Inquire of AsyncAgent     // Check the status
and AuctionReply =
  | StartBidding
  | Status of int * DateTime // Asked sum and expiration
  | BestOffer                // Ours is the best offer
  | BeatenOffer of int       // Yours is beaten by another offer
  | AuctionConcluded of      // Auction concluded
      AsyncAgent * AsyncAgent
  | AuctionFailed            // Failed without any bids
  | AuctionOver              // Bidding is closed
let timeToShutdown = 3000
let bidIncrement = 10

This is the format of the messages that the clients can send and the action agent can reply to. F# is really good at this sort of thing. First, we need an auction agent:

let auctionAgent seller minBid closing =
    let agent = spawnAgent (fun msg (isConcluded, maxBid, maxBidder) ->
                            match msg with
                            | Offer (_, client) when isConcluded ->
                                client <-- AuctionOver
                                (isConcluded, maxBid, maxBidder)
                            | Offer(bid, client) when not(isConcluded) ->
                                if bid >= maxBid + bidIncrement then
                                    if maxBid >= minBid then maxBidder <-- BeatenOffer bid
                                    client <-- BestOffer
                                    (isConcluded, bid, client)
                                else
                                    client <-- BeatenOffer maxBid
                                    (isConcluded, maxBid, maxBidder)
                            | Inquire client    ->
                                client <-- Status(maxBid, closing)
                                (isConcluded, maxBid, maxBidder))
                            (false, (minBid - bidIncrement), spawnWorker (fun _ -> ()))

Notice that, if the action is concluded, the agent replies to offers by sending an AuctionOver message. If the auction is still open, then, in case the bid is higher than the max, it sets a new max and notify the two parties involved; otherwise it notifies the bidder that the offer wasn’t successful. Also you can ask for the status of the auction.

This is what the code above says. Maybe the code is simpler than words. Anyhow, we need to treat the case where no message is received for some amount of time.

agent <-- SetTimeoutHandler
            (closing - DateTime.Now).Milliseconds
            (fun (isConcluded: bool, maxBid, maxBidder) ->
                if maxBid >= minBid then
                  let reply = AuctionConcluded(seller, maxBidder)
                  maxBidder <-- reply
                  seller <-- reply
                else seller <-- AuctionFailed
                agent <-- SetTimeoutHandler
                    timeToShutdown
                    (fun (_:bool, _:int,_:AsyncAgent) -> StopProcessing)
                ContinueProcessing (true, maxBid, maxBidder))
agent

We start by waiting for the amount of time to the closing of the auction. If we get no messages, then two things might happen: we have an offer that is more than the minimum or we don’t. If we do, we tell everyone that it’s finished. Otherwise, we tell the seller that its item wasn’t successful. In any case, we prepare the agent to shutdown by setting its next timeout to be timeoutToShutdown.

It is interesting that we set the timeout handler inside the timeout handler. This is not a problem because of the nature of message processing (aka it processes one message at the time).

We then need a bunch of of symbols …

module Auction =
  let random = new Random()
  let minBid = 100
  let closing = DateTime.Now.AddMilliseconds 10000.
  let seller = spawnWorker (fun (msg:AuctionReply) -> ())
  let auction = auctionAgent seller minBid closing

Not a very smart seller we have here … Next up is our definition of a client.

let rec c = spawnAgent (
                fun msg (max, current) ->
                    let processBid (aMax, aCurrent) =
                        if aMax >= top then
                            log "too high for me"
                            (aMax, aCurrent)
                        elif aCurrent < aMax then
                              let aCurrent = aMax + increment
                              Thread.Sleep (1 + random.Next 1000)
                              auction <-- Offer(aCurrent, c)
                              (aMax, aCurrent)
                        else (aMax, aCurrent)
                    match msg with
                    | StartBidding      ->
                        auction <-- Inquire c
                        (max, current)
                    | Status(maxBid,_)  ->
                        log <| sprintf "status(%d)" maxBid
                        let s = processBid (maxBid, current)
                        c <-- SetTimeoutHandler timeToShutdown (fun _ -> StopProcessing)
                        s
                    | BestOffer ->
                        log <| sprintf "bestOffer(%d)" current
                        processBid(max, current)
                    | BeatenOffer maxBid ->
                        log <| sprintf "beatenOffer(%d)" maxBid
                        processBid(maxBid, current)
                    | AuctionConcluded(seller, maxBidder) ->
                        log "auctionConcluded"
                        c <-- Stop
                        (max, current)
                    | AuctionOver ->
                        log "auctionOver"
                        c <-- Stop
                        (max, current))
                 (0,0)
c

Something that I like about agents is the fact that you need to understand just small snippets of code at the time. For example, you can read the processing for BestOffer and figure out if it makes sense. I have an easy time personalizing them as in : “Ok, the guy just got a notification that there has been a better offer, what is he going to do next?”.

The code should be self explanatory for the most part. In essence, if you can offer more, do it otherwise wait for the auction to end. I’m not even sure the processing is completely right. I confess I’m just trying to do the same as Matthews code from the link above.

We can then start up the whole thing and enjoy the cool output.

open Auction
(client 1 20 200) <-- StartBidding
(client 2 10 300) <-- StartBidding
(client 3 30 150) <-- StartBidding
Console.ReadLine() |> ignore

Now for the nasty part. Implementing the framework.

LAgent: an agent framework in F# – Part VI – Hot swapping of code (and something silly)

Fri, 03 Jul 2009 15:23:00 GMT

Download framework here.

All posts are here:

Hot swapping of code

Let’s get back a couple of steps and consider what happens when you get an error. Sure, your agent will continue processing messages, but it might be doing the wrong thing. Your message handling code might be buggy.

Ideally you’d want to patch things on the fly. You’d want to replace the message processing code for an agent without stopping it.

Here is how you do it:

let counter2 = spawnAgent (fun msg state -> printfn "From %i to %i" state (state + msg);
                                                                              state + msg) 0
counter2 <-- 2
counter2 <-- SetAgentHandler(fun msg state ->
                printfn "From %i to %i via multiplication" state (state * msg); msg * state)
counter2 <-- 3

Which generates:

**From 0 to 2

From 2 to 6 via multiplication**

After the agent receives a SetAgentHandler message, it switch from a ‘+’ agent to a ‘*’ agent on the fly!! All the messages that come after that one gets multiplied to the state. Also, the state is preserved between changes in behavior.

It might not be immediately apparent how to load a function at runtime, but it is really simple. Imagine that I get the data on the function to load from somewhere (i.e. a management console UI).

let assemblyNameFromSomewhere, typeNameFromSomewhere, methodNameFromSomewhere = 
                                                "mscorlib.dll", "System.Console", "WriteLine"

I can then use it to dynamically load a message handler (in this case Console.Writeline).

let a = Assembly.Load(assemblyNameFromSomewhere)
let c = a.GetType(typeNameFromSomewhere)
let m = c.GetMethod(methodNameFromSomewhere, [|"".GetType()|])
let newF = fun (msg:string) (state:obj) -> m.Invoke(null, [| (msg:>obj) |])

And then it is as simple as posting a SetAgentHandler.

counter2 <-- SetAgentHandler(newF)
counter2 <-- "blah"

Now our counter2 agent has become an echo agent on the fly, having loaded Console.WriteLine dynamically. Note how the agent moved from being a ‘+’ agent taking integers to being a ‘*’ agent taking integers to being an ‘echo’ agent taking strings. And it didn’t stop processing messages for the whole time.

Obviously, you can do the same thing with workers:

echo <-- SetWorkerHandler(fun msg -> printfn "I'm an echo and I say: %s" msg)
echo <-- "Hello"

And parallelWorkers:

parallelEcho <-- SetWorkerHandler(fun msg -> tprint ("I'm new and " + msg))
messages |> Seq.iter (fun msg -> parallelEcho <-- msg)

A silly interlude

As a way to show some agents talking to each other, here is a simple program that simulates marital interactions (of the worst kind):

let rec husband = spawnWorker (fun (To, msg) -> printfn "Husband says: %s" msg; To <-- msg)
let rec wife = spawnWorker (fun msg -> printfn "Wife says: screw you and your '%s'" msg)
husband <-- (wife, "Hello")
husband <-- (wife, "But darling ...")
husband <-- (wife, "ok")

Which produces:

**Husband says: Hello

Husband says: But darling

Wife says: screw you and your ‘Hello’

Wife says: screw you and your ‘But darling’

Husband says: ok

Wife says: screw you and your ‘ok’**

And yes, you cannot expect messages to be in the right sequence … Next up is an auction application.

LAgent: an agent framework in F# – Part V – Timeout management

Fri, 26 Jun 2009 14:56:00 GMT

Download framework here.

All posts are here:

Timeout management

Timeouts are very important in message based systems. In essence, if you are not getting messages for a certain period of time, that usually means something. It might be that something crashed, that other agents think that you are not online, or any other number of things. Hence the need to set timeouts and react when they are triggered.

You do that by writing the following:

counter1 <--SetTimeoutHandler 1000 
            (fun state -> printfn "I'm still waiting for a message in state %A, come on ..." 
                                                            state; ContinueProcessing(state))

Which generates the following message every second:

**I’m still waiting for a message in state 2, come on

I’m still waiting for a message in state 2, come on .…**

The first parameter to SetTimeoutHandler is how long to wait before triggering the handler. The second parameter is the handler that gets called whenever no message is received for that amount of time. Notice that the handler takes the current state of the agent and returns ContinueProcessing(state). This tells the agent to continue processing messages and sets the current state to state.

The following code:

counter1 <-- 2

Then generates:

**I’m still waiting for a message in state 4, come on

I’m still waiting for a message in state 4, come on**

ContinueProcessing is just one of the three possible values of the (terribly named) AfterError:

type AfterError =
| ContinueProcessing of obj
| StopProcessing
| RestartProcessing

Let’s see what RestartProcessing does.

counter1 <-- SetTimeoutHandler 1000  (fun state -> printfn "Restart from state %A" state
                                                                        ; RestartProcessing)

Which, as expected, generates a nice stream of:

**Restart from state 0

Restart from state 0**

To bring things back to normal (aka no timeout) you can just pass –1 as in:

counter1 <-- SetTimeoutHandler -1  (fun state -> ContinueProcessing(state))

Also, you can stop the agent when a timeout occurs by returning the aptly named StopProcessing:

counter1 <-- SetTimeoutHandler 1000  (fun state -> printfn "Restart from state %A" state; 
                                                                             StopProcessing)

Another interesting thing you might want to do is hot swapping of code. More on that in the next part …

LAgent: an agent framework in F# – Part IV – Custom error management

Fri, 19 Jun 2009 14:36:00 GMT

Download framework here.

All posts are here:

Custom error management

In the last part we saw what happens by default in the framework when an error occurs. But that might not be what you want. You might want to have your sophisticated error detection and recovery distributed algorithm.

To make such a thing possible each agent has a manager. The manager is an agent that gets called whenever an error occurs in the agent it is monitoring.

In code:

let manager = spawnWorker (fun (agent, name:string, ex:Exception, msg:obj,
                 state, initialState) -> printfn "%s restarting ..." name; agent <-- Restart)
counter1 <-- SetManager(manager)

Whenever an error is generated the manager receives a tuple of:

(agent, name, exception, message, currentState, inititialState)

This manager prints out something and then restarts the agent. Let’s trigger an error by posting the wrong message:

counter1 <-- "afdaf"
counter1 <-- 2

The expectation is that the counter will restart from 0 whenever an error is triggered. This is what happens:

**Bob restarting

From 0 to 2**

Which is what we expected. Obviously this is not a very sophisticated error recovery algorithm. You might want to do something more meaningful. Hopefully you have enough information to build whatever you need.

A particularly important class of unexpected event is timeouts. We’ll talk about them next.

LAgent: an agent framework in F# – Part III – Default error management

Fri, 12 Jun 2009 14:07:00 GMT

Download framework here.

All posts are here:

Default error management

What happens when an error occurs? Well, ideally you want to notify someone and continue processing messages. By default you want to print the error and as much information as you can about it.

Let’s first see what happens if you pass the wrong message type:

counter1 <-- "fst"

Generates:

**> The exception below occurred on agent Undefined at state 3 with message “fst”. The agent was started with state 0.

System.InvalidCastException: Specified cast is not valid.

   at Microsoft.FSharp.Core.LanguagePrimitives.IntrinsicFunctions.UnboxGeneric[T](Object source)

   at FSI_0003.AgentSystem.f@158.Invoke(Object a, Object b)

   at Microsoft.FSharp.Core.FastFunc`2.InvokeFast[V](FastFunc`2 func, T arg1, TResult arg2)

   at Microsoft.FSharp.Core.FastFunc`2.InvokeFast[V](FastFunc`2 func, T arg1, TResult arg2)

   at FSI_0003.AgentSystem.loop@20-3.Invoke(Unit unitVar)

   at Microsoft.FSharp.Control.AsyncBuilderImpl.callA@245.Invoke(AsyncParams`1 args)**

You get information about the current state of the agent, the message that generated the error, the initial state of the agent and the exception that was generated. But, in a system with several agents, you’d like to know which one agent failed. Then you need to name your agent:

counter1 <-- SetName("Bob")
counter1 <-- "fadfad"

Now you get (important part in blue):

**> The exception below occurred on agent Bob at state 3 with message “fadfad”. The agent was started with state 0.

System.InvalidCastException: Specified cast is not valid.

   at Microsoft.FSharp.Core.LanguagePrimitives.IntrinsicFunctions.UnboxGeneric[T](Object source)

   at FSI_0003.AgentSystem.f@158.Invoke(Object a, Object b)

   at Microsoft.FSharp.Core.FastFunc`2.InvokeFast[V](FastFunc`2 func, T arg1, TResult arg2)

   at Microsoft.FSharp.Core.FastFunc`2.InvokeFast[V](FastFunc`2 func, T arg1, TResult arg2)

   at FSI_0003.AgentSystem.loop@20-3.Invoke(Unit unitVar)

   at Microsoft.FSharp.Control.AsyncBuilderImpl.callA@245.Invoke(AsyncParams`1 args)**

The important thing is that the agent continues running. It lives to fight another day. Hence:

counter1 <-- 3

Produces:

From 3 to 6

Which shows that the agent is running and that it has kept its current state. Also errors can occur inside the message handler with a similar result:

(spawnAgent (fun msg state -> state / msg) 100) <-- 0

Produces:

**> The exception below occurred on agent Undefined at state 100 with message 0. The agent was started with state 100.

System.DivideByZeroException: Attempted to divide by zero.

   at FSI_0013.it@48-3.Invoke(Int32 msg, Int32 state)

   at Microsoft.FSharp.Core.FastFunc`2.InvokeFast[V](FastFunc`2 func, T arg1, TResult arg2)

   at FSI_0003.AgentSystem.f@158.Invoke(Object a, Object b)

   at Microsoft.FSharp.Core.FastFunc`2.InvokeFast[V](FastFunc`2 func, T arg1, TResult arg2)

   at Microsoft.FSharp.Core.FastFunc`2.InvokeFast[V](FastFunc`2 func, T arg1, TResult arg2)

   at FSI_0003.AgentSystem.loop@20-3.Invoke(Unit unitVar)

   at Microsoft.FSharp.Control.AsyncBuilderImpl.callA@245.Invoke(AsyncParams`1 args)**

But this might not be what you want. You might want to customize what happens when an error occurs. We’ll talk about that next.

LAgent : an agent framework in F# – Part II – Agents and control messages

Fri, 05 Jun 2009 14:03:00 GMT

Download framework here.

All posts are here:

Agents

Agents are entities that process messages and keep state between one message and the next. As such they need to be initialized with a lambda that takes a message and a state and returns a new state. In F# pseudo code: msg –> state –> newState. For example the following:

let counter = spawnAgent (fun msg state -> state + msg) 0

This is a counter that starts from 0 and gets incremented by the value of the received message. Let’s make it print something when it receives a message:

let counter1 = spawnAgent
                (fun msg state -> printfn "From %i to %i" state (state + msg); state + msg) 0
counter1 <-- 3
counter1 <-- 4

Which produces:

**From 0 to 3

From 3 to 7**

There is no spawnParallelAgent, because I couldn’t figure out its usage patterns. Maybe I don’t have enough creativity. Obviously msg and state could be of whatever type (in real application they end up being tuples more often than not).

Control messages

You can do things to agents. I’m always adding to them but at this stage they are:

type Command =
| Restart
| Stop
| SetManager of AsyncAgent
| SetName of string

Plus some others. I’ll describe most of them later on, right now I want to talk about Restart and Stop. You use the former like this:

counter1 <-- Restart
counter1 <-- 3

Which produces:

From 0 to 3

This should be somehow surprising to you. You would have thought that you could just post integers to a counter. This is not the case. You can post whatever object. This is useful because it allows to have a common model for passing all sort of messages, it allows for the agent not to be parameterized by the type of the message (and of state) so that you can store them in data structures and allows advanced scenarios (i.e. hot swapping of code).

This is a debatable decision. I tried to get the best of strongly typing and dynamic typing, while keeping simplicity of usage. The implementation of this is kind of a mess though. We’ll get there.

BTW: you use Stop just by posting Stop, which stops the agent (forever).

LAgent : an agent framework in F# – Part I – Workers and ParallelWorkers

Fri, 29 May 2009 13:26:00 GMT

Download framework here.

All posts are here:

Introduction

I like to try out different programming paradigms. I started out as an object oriented programmer. In university, I used Prolog. I then learned functional programming. I also experimented with various shared memory parallel paradigms (i.e. async, tasks and such). I now want to learn more about message based parallel programming (Erlang style). I’m convinced that doing so makes me a better programmer. Plus, I enjoy it …

My usual learning style is to build a framework that replicates a particular programming model and then write code using it. In essence, I build a very constrained environment. For example, when learning functional programming, I didn’t use any OO construct for a while even if my programming language supports them.

In this case, I built myself a little agent framework based on F# MailboxProcessors. I could have used MailboxProcessors directly, but they are too flexible for my goal. Even to write a simple one of these guys, you need to use async and recursion in a specific pattern, which I always forget. Also, there are multiple ways to to do Post. I wanted things to be as simple as possible. I was willing to sacrifice flexibility for that.

Notice that there are serious efforts in this space (as Axum). This is not one of them. It’s just a simple thing I enjoy working on between one meeting and the next.

Workers and ParallelWorkers

The two major primitives are spawning an agent and posting a message.

let echo = spawnWorker (fun msg -> printfn "%s" msg)
echo <-- "Hello guys!"

There are two kinds of agents in my system. A worker is an agent that doesn’t keep any state between consecutive messages. It is a stateless guy. Notice that the lambda that you pass to create the agent is strongly typed (aka msg is of type string). Also notice that I overloaded the <— operator to mean Post.

Given that a worker is stateless, you can create a whole bunch of them and, when a message is posted, route it to one of them transparently.

let parallelEcho = spawnParallelWorker(fun s -> printfn "%s" s) 10
parallelEcho <-- "Hello guys!"

For example, in the above code, 10 workers are created and, when a message is posted, it gets routed to one of them (using a super duper innovative dispatching algorithm I’ll describe in the implementation part). This parallelWorker guy is not really needed, you could easily built it out of the other primitives, but it is kind of cute.

To show the difference between a worker and a parallelWorker, consider this:

let tprint s = printfn "%s running on thread %i" s Thread.CurrentThread.ManagedThreadId
let echo1 = spawnWorker (fun s -> tprint s)
let parallelEcho1 = spawnParallelWorker(fun s -> tprint s) 10
let messages = ["a";"b";"c";"d";"e";"f";"g";"h";"i";"l";"m";"n";"o";"p";"q";"r";"s";"t"]
messages |> Seq.iter (fun msg -> echo1 <-- msg)
messages |> Seq.iter (fun msg -> parallelEcho1 <-- msg)

The result of the echo1 iteration is:

**a running on thread 11

b running on thread 11

c running on thread 11

d running on thread 11

…**

While the result of the parallelEcho1 iteration is:

**a running on thread 13

c running on thread 14

b running on thread 12

o running on thread 14

m running on thread 13

…**

Notice how the latter executes on multiple threads (but not in order). Next time I’ll talk about agents, control messages and error management.

Luke talks about F# on Channel9

Wed, 27 May 2009 12:47:00 GMT

Here you go: https://channel9.msdn.com/posts/Charles/Luke-Hoban-Latest-version-of-F-Released-Whats-the-story-Whats-next/

A version of the AsyncCache found its way into the Parallel Programming samples …

Thu, 21 May 2009 13:28:13 GMT

Go here to download them. It is in ParallelExtensionsExtrasCoordinationDataStructures. It has a slightly different design in that it returns Tasks. I’m trying to get Stephen to blog about it so that you can compare them.

I talk about C# and VB Co-Evolution on Channel9 (and some F# …)

Thu, 14 May 2009 13:26:18 GMT

The title says it all. If you are interested, go here.

An Async Html cache – part II – Testing the cache

Fri, 08 May 2009 11:34:00 GMT

Other posts:

Part I – Writing the cache

Let’s try out our little cache. First I want to write a synchronous version of it as a baseline.

Private Shared Sub TestSync(ByVal sites() As String, ByVal sitesToDownload As Integer, ByVal howLong As Integer)
    Dim syncCache As New Dictionary(Of String, String)
    Dim count = sites.Count()
    Dim url1 = "http://moneycentral.msn.com/investor/invsub/results/statemnt.aspx?Symbol="
    For i = 0 To sitesToDownload - 1
        Dim html As String = ""
        Dim url = url1 & sites(i Mod count)
        If Not syncCache.TryGetValue(url, html) Then
            html = LoadWebPage(url)
            syncCache(url) = html
        End If
        DoWork(html, howLong)
    Next
End Sub

This is a loop that loads webpages in the cache if they are not already there. sites is a list of tickers used to compose the urls; sitesToDownload is the total number of sites to download, so that a single url can be loaded multiple times; howLong represents the work to be done on each loaded page.

In this version the cache is simply a Dictionary and there is no parallelism. The two bold lines is where the cache is managed.

DoWork is this.

Public Shared Sub DoWork(ByVal html As String, ByVal howLong As Integer)
    Thread.Sleep(howLong)
End Sub

Let’s take a look at the asynchronous version.

Private Shared Sub TestAsync(ByVal sites() As String, ByVal sitesToDownload As Integer, ByVal howLong As Integer)
    Dim htmlCache As New HtmlCache
    Dim count = sites.Count()
    Dim url = "http://moneycentral.msn.com/investor/invsub/results/statemnt.aspx?Symbol="
    Using ce = New CountdownEvent(sitesToDownload)
        For i = 1 To sitesToDownload
            htmlCache.GetHtmlAsync(
                url & sites(i Mod count),
                Sub(s)
                    DoWork(s, howLong)
                    ce.Signal()
                End Sub)
        Next
        ce.Wait()
    End Using

There are several points worth making on this:

The lambda used as second parameter for GetHtmlAsync is invoked on a different thread whenever the html has been retrieved (which could be immediately if the cache has downloaded the url before)
CountDownEvent allows a thread to wait for a certain number of signals to be sent. The waiting happens on the main thread in the ce.Wait() instruction. The triggering of the event happens in the lambda described in the point above (the ce.Signal() instruction)

This is the driver for the overall testing.

Private Shared Sub TestPerf(ByVal s As String, ByVal a As Action, ByVal iterations As Integer)
    Dim clock As New Stopwatch
    clock.Start()
    For i = 1 To iterations
        a()
    Next
    clock.Stop()
    Dim ts = clock.Elapsed
    Dim elapsedTime = String.Format(s & ": {0:00}:{1:00}:{2:00}.{3:00}", ts.Hours, ts.Minutes, ts.Seconds, ts.Milliseconds / 10)
    Console.WriteLine(elapsedTime, "RunTime")
End Sub

There is not much to say about it. Start the clock, perform a bunch of iterations of the passed lambda, stop the clock, print out performance.

And finally the main method. Note that all the adjustable parameters are factored out before the calls to TestPerf.

Public Shared Sub Main()
    Dim tickers = New String() {"mmm", "aos", "shlm", "cas", "abt", "anf", "abm", "akr", "acet", "afl", "agl", "adc", "apd",
                               "ayr", "alsk", "ain", "axb", "are", "ale", "ab", "all"}
    Dim sitesToDownload = 50
    Dim workToDoOnEachUrlInMilliSec = 20
    Dim perfIterations = 5
    TestPerf("Async", Sub() TestAsync(tickers, sitesToDownload, workToDoOnEachUrlInMilliSec), perfIterations)
    TestPerf("Sync", Sub() TestSync(tickers, sitesToDownload, workToDoOnEachUrlInMilliSec), perfIterations)
End Sub

Feel free to change (tickers, sitesToDownload, workToDoOnEachUrlInMilliSec, perfIterations). Depending on the ratios between these parameters and the number of cores on your machine, you’re going to see different results. Which highlights the fact that parallelizing your algorithms can yield performance gains or not depending on both software and hardware considerations. I get ~3X improvement on my box. I attached the full source file for your amusement.

AsyncCache.vb

An Async Html cache – Part I - Writing the cache

Mon, 27 Apr 2009 19:57:00 GMT

Other posts:

Part II - Testing the cache

In the process of converting a financial VBA Excel Addin to .NET (more on that in later posts), I found myself in dire need of a HTML cache that can be called from multiple threads without blocking them. Visualize it as a glorified dictionary where each entry is (url, cachedHtml). The only difference is that when you get the page, you pass a callback to be invoked when the html has been loaded (which could be immediately if the html had already been retrieved by someone else).

In essence, I want this:

Public Sub GetHtmlAsync(ByVal url As String, ByVal callback As Action(Of String))

I’m not a big expert in the .Net Parallel Extensions, but I’ve got help. Stephen Toub helped so much with this that he could have blogged about it himself. And, by the way, this code runs on Visual Studio 2010, which we haven’t shipped yet. I believe with some modifications, it can be run in 2008 + .Net Parallel Extensions CTP, but you’ll have to change a bunch of names.

In any case, here it comes. First, let’s add some imports.

Imports System.Collections.Concurrent
Imports System.Threading.Tasks
Imports System.Threading
Imports System.Net

Then, let’s define an asynchronous cache.

Public Class AsyncCache(Of TKey, TValue)

This thing needs to store the (url, html) pairs somewhere and, luckily enough, there is an handy ConcurrentDictionary that I can use. Also the cache needs to know how to load a TValue given a TKey. In ‘programmingese’, that means.

Private _loader As Func(Of TKey, TValue)
Private _map As New ConcurrentDictionary(Of TKey, Task(Of TValue))

I’ll need a way to create it.

Public Sub New(ByVal l As Func(Of TKey, TValue))
    _loader = l
End Sub

Notice in the above code the use of the Task class for my dictionary instead of TValue. Task is a very good abstraction for “do some work asynchronously and call me when you are done”. It’s easy to initialize and it’s easy to attach callbacks to it. Indeed, this is what we’ll do next:

Public Sub GetValueAsync(ByVal key As TKey, ByVal callback As Action(Of TValue))
    Dim task As Task(Of TValue) = Nothing
    If Not _map.TryGetValue(key, task) Then
        task = New Task(Of TValue)(Function() _loader(key), TaskCreationOptions.DetachedFromParent)
        If _map.TryAdd(key, task) Then
            task.Start()
        Else
            task.Cancel()
            _map.TryGetValue(key, task)
        End If
    End If
    task.ContinueWith(Sub(t) callback(t.Result))
End Sub

Wow. Ok, let me explain. This method is divided in two parts. The first part is just a thread safe way to say “give me the task corresponding to this key or, if the task hasn’t been inserted in the cache yet, create it and insert it”. The second part just says “add callback to the list of functions to be called when the task has finished running”.

The first part needs some more explanation. What is TaskCreationOptions.DetachedFromParent? It essentially says that the created task is not going to prevent the parent task from terminating. In essence, the task that created the child task won’t wait for its conclusion. The rest is better explained in comments.

If Not _map.TryGetValue(key, task) Then ' Is the task in the cache? (Loc. X)
    task = New Task(Of TValue)(Function() _loader(key), TaskCreationOptions.DetachedFromParent) ' No, create it
    If _map.TryAdd(key, task) Then ' Try to add it
        task.Start() ' I succeeded. I'm the one who added this task. I can safely start it.
    Else
        task.Cancel() ' I failed, someone inserted the task after I checked in (Loc. X). Cancel it.
        _map.TryGetValue(key, task) ' And get the one that someone inserted
    End If
End If

Got it? Well, I admit I trust Stephen that this is what I should do …

I can then create my little HTML Cache by using the above class as in:

Public Class HtmlCache
    Public Sub GetHtmlAsync(ByVal url As String, ByVal callback As Action(Of String))
        _asyncCache.GetValueAsync(url, callback)
    End Sub
    Private Function LoadWebPage(ByVal url As String) As String
        Using client As New WebClient()
            'Test.PrintThread("Downloading on thread {0} ...")
            Return client.DownloadString(url)
        End Using
    End Function
    Private _asyncCache As New AsyncCache(Of String, String)(AddressOf LoadWebPage)
End Class

I have no idea why coloring got disabled when I copy/paste. It doesn’t matter, this is trivial. I just create an AsyncCache and initialize it with a method that knows how to load a web page. I then simply implement GetHtmlAsync by delegating to the underlying GetValueAsync on AsyncCache.

It is somehow bizarre to call Webclient.DownloadString, when the design could be revised to take advantage of its asynchronous version. Maybe I’ll do it in another post. Next time, I’ll write code to use this thing.

Luca at NDC in Oslo 17 – 19 June 2009

Tue, 24 Mar 2009 12:46:27 GMT

I’ll be speaking about the future of C# and F#. Oslo brings back so many memories …

“one of the world’s most important conferences for IT developers and leaders”

Excel Financial functions 2.0 released

Wed, 25 Feb 2009 14:55:00 GMT

I simply fixed a bug related to the Rate function (wrong name for parameters).

It is here: http://code.msdn.microsoft.com/FinancialFunctions

Simulating INumeric with dynamic in C# 4.0

Thu, 05 Feb 2009 11:47:26 GMT

When I wrote my Excel financial library I agonized over the decision of which numeric type to use to represent money. Logic would push me toward decimal, but common usage among financial library writers would push me toward double. I ended up picking double, but I regret having to make that choice in the first place.

Conceptually, I’d like my numeric functions to work for anything that supports the basic arithmetic operators (i.e. +, -, *). Unfortunately that is not possible in .NET at this point in time. In essence you have to write your code twice as below.

static double SumDouble(double a, double b) { return a + b; }
static decimal SumDecimal(decimal a, decimal b) { return a + b; }

Granted, this is not a good state of affairs. We often discussed how to make it work, but we couldn’t find a solution that was both fast to run and cheap for us to implement. More often than not we speculated about having the numeric types implement a specific INumeric interface and add a generic constraint to the C#/VB languages to make it work. Hence the title of this post.

With we implemented dynamic in C# 4.0 it occurred to me that you can fake your way into writing your code just once. For sure, this solution doesn’t have the same performance characteristics of ‘writing your code twice’, but at least it doesn’t duplicate your code.

This is how it looks like:

static dynamic Sum1(dynamic a, dynamic b) { return a + b; }

The call to the ‘+’ operator is resolved at runtime, by the C# binder, hence a performance penalty is incurred. The penalty is less than you might think, given that the DLR caches things under the cover so that no v-table lookup is performed the second time around. The whole thing is explained in more detail here. But still, it is not as fast as a normal ‘+’ operator over a primitive type. I’ll let you enjoy micro performance testing this one 🙂

A slight refinement is to make the code generic so that a caller doesn’t see a signature with dynamic types as arguments.

static dynamic Sum2<T1, T2>(T1 a, T2 b)
{
    dynamic ad = a;
    dynamic bd = b;
    return ad + bd;
}

I could make the return type generic as well, but that would force the caller to be explicit about the types, making the calling code much less readable. The other good thing about this signature is that you get a different call site with each combination of type arguments and, since they are separate, the binding caches should stay small. With the former signature there is only one call site and the cache could pile up to the point where the DLR decides to discard it.

Here is how the calling code looks like right now:

Console.WriteLine(Sum2(2m, 4m));
Console.WriteLine(Sum2(2.0, 4.0));
Console.WriteLine(Sum2(new DateTime(2000,12,1), new TimeSpan(24,0,0)));

Yet another way to write this code is as follows:

public static T Sum3<T>(T a, T b)
{
    dynamic ad = a;
    dynamic bd = b;
    return ad + bd;
}

This gets around the problem of showing a dynamic return value and give you some more compile time type checking. But it prevents summing not coercible types. The compiler doesn’t let you get there. The last line below wont’ compile:

Console.WriteLine(Sum3(2m, 4m));
Console.WriteLine(Sum3(2.0, 4.0));
//Console.WriteLine(Sum3(new DateTime(2000,12,1), new TimeSpan(24,0,0)));

Also notice that in VB you could have done this a long time ago 🙂

Function Sum(Of T1, T2)(ByVal a As T1, ByVal b As T2)
    Dim aa As Object = a
    Dim bb As Object = b
    Return aa + bb
End Function

In summary, by using dynamic you can write your numeric code just once, but you pay a performance price. You are the only one who can decide if the price is worth paying in your particular application. As often, the profiler is your friend.

New release of Financial Functions .NET uploaded on MSDN Code Gallery

Tue, 27 Jan 2009 14:18:00 GMT

I fixed the bug described in this thread and cleaned up the root finding algorithm. I’m still unhappy about it, but I have no time to code a better one right now (i.e. Ridder, Brent). I also added changes.txt and todo.txt to keep track of things.

Changes.txt

V1
1. Fixed call to throw in bisection
2. Changed findBounds algo
3. Added TestXirrBugs function
4. Removed the NewValue functions everywhere

ToDo.txt

The interaction of Bisection and Newton algo in findRoot needs review. It seems like it is working now, but it could use some love. Maybe I should switch to a better root finding algo (i.e. Rudder or Brent)

Bayesian inference in F# – Part IIb – Finding Maia underlying attitude

Mon, 19 Jan 2009 11:48:00 GMT

Other parts:

The previous post ended on this note.

let MaiaJointProb attitude action =
    match attitude with
    | Happy     -> happyActions |> List.assoc action
    | UnHappy   -> unHappyActions |> List.assoc action
    | Quiet     -> quietActions |> List.assoc action

This is just a two by two matrix. It simply represents which probability is associated to an (attitude, action) tuple. It is useful to think about it in these terms, because it makes easier to grasp the following function:

/// Conditional probability of a mental state, given a particular observed action
let MaiaLikelihood action = fun attitude -> MaiaJointProb attitude action

This is simply a row in the matrix. It answers the question: given that I observe a particular action, what is the probability that Maia has a certain attitude?. This is called “likelihood function” in statistics. Its general form is: given that a I observe an outcome, what is the probability that it is generated by a process with a particular parameter?

A related question is then: what if I observe a sequence of independent actions? What is the probability that the baby has a certain attitude then? This is answered by the following:

/// Multiple applications of the previous conditional probabilities for a series of actions (multiplied)
let MaiaLikelihoods actions =
    let composeLikelihoods previousLikelihood action  = fun attitude -> previousLikelihood attitude * MaiaLikelihood action attitude
    actions |> Seq.fold composeLikelihoods (fun attitude -> 1.)

It is a trivial extension of the previous function (really), once you know that to combine likelihoods you multiply them.

We now need to describe what our prior is. A prior is our preconceived notion about a particular parameter (in this case the baby’s attitude). You might be tempted to express that notion with a single value, but that would be inaccurate. You need to indicate how confident you are about it. In statistics you do that by choosing a distribution for your belief. This is one of the beauties of Bayesian statistics, everything is a probability distribution. In this case, we really don’t have any previous belief, so we pick the uniform distribution.

let MaiaUniformPrior attitude = 1. / 3.

Think of this as: you haven’t read any baby-attitude-specific study or received any external information about the likely attitude of Maia, so you cannot prefer one attitude over another.

We are almost done. Now we have to apply the Bayesian theorem and get the un-normalized posterior distribution. Forget about the un-normalized word. What is a posterior distribution? This is your output, your return value. It says: given my prior belief on the value of a parameter and given the outcomes that I observed, this is what I now believe the parameter to be. In this case it goes like: I had no opinion on Maia’s attitude to start with, but after I observed her behavior for a while, I now think she is Happy with probability X, UnHappy with probability Y and Quiet with probability Z.

/// Calculates the unNormalized posterior given prior and likelihood
let unNormalizedPosterior (prior:'a -> float) likelihood =
    fun theta -> prior theta * likelihood theta

We then need to normalize this thing (it doesn’t sum to one). The way to do it is to divide each probability by the sum of the probabilities for all the possible outcomes.

/// All possible values for the unobservable parameter (mental state)
let support = [Happy; UnHappy; Quiet]
/// Normalize the posterior (it integrates to 1.)
let posterior prior likelihood =
    let post = unNormalizedPosterior prior likelihood
    let sum = support |> List.sum_by (fun attitude -> post attitude)
    fun attitude -> post attitude / sum

We are done. Now we can now start modeling scenarios. Let’s say that you observe [Smile;Smile;Cry;Smile;LookSilly]. What could the underlying attitude of Maia be?

let maiaIsANormalBaby = posterior MaiaUniformPrior (MaiaLikelihoods [Smile;Smile;Cry;Smile;LookSilly])

We can then execute our little model:

maiaIsANormalBaby Happy
maiaIsANormalBaby UnHappy
maiaIsANormalBaby Quiet

And we get (0.5625, 0.0625, 0.375). So Maia is likely to be happy and unlikely to be unhappy. Let’s now model one extreme case:

/// Extreme cases
let maiaIsLikelyHappyDist = posterior MaiaUniformPrior (MaiaLikelihoods [Smile;Smile;Smile;Smile;Smile;Smile;Smile])
maiaIsLikelyHappyDist Happy
maiaIsLikelyHappyDist UnHappy
maiaIsLikelyHappyDist Quiet

And we get (0.944, 0.000431, 0.05). Now Maia is almost certainly Happy. Notice that I can confidently make this affirmation because my end result is exactly what I was looking for when I started my quest. Using classical statistics, that wouldn’t be the case.

A related question I might want to ask is: given the posterior distribution for attitude that I just found, what is the probability of observing a particular action? In other words, given the model that I built, what does it predict?

let posteriorPredictive jointProb posterior =
    let composeProbs previousProbs attitude = fun action -> previousProbs action + jointProb attitude action * posterior attitude
    support |> Seq.fold composeProbs (fun action -> 0.)
let nextLikelyUnknownActionDist = posteriorPredictive MaiaJointProb maiaIsLikelyHappyDist

I don’t have the strength right now to explain the mathematical underpinning of this. In words, this says: considering that Maia can have one of the possible three Attitudes with the probability calculated above, what is the probability that I observe a particular action? Notice that the signature for it is: (Action -> float), which is the compiler way to say it.

Now we can run the thing.

nextLikelyUnknownActionDist Smile
nextLikelyUnknownActionDist Cry
nextLikelyUnknownActionDist LookSilly

And we get (0.588, 0.2056, 0.2055). Why is that? We’ll talk about it in the next post.

Financial Functions for .NET released !

Thu, 04 Dec 2008 16:40:00 GMT

Today I released the following library on CodeGallery. It is the result of three months of coding during my paternity leave in Italy. You can get it from here.

What is it?
This is a .NET library that provides the full set of financial functions from Excel. The main goal for the library is compatibility with Excel, by providing the same functions, with the same behaviour. Note though that this is not a wrapper over the Excel library; the functions have been re-implemented in managed code so that you do not need to have Excel installed to use this library.

Where I can find documentation on these functions?
Just open Excel and click on Formulas/Financial or go to this

I don’t think one of the function is right. Excel produces the wrong results! Why don’t you do it right?
My goal is to replicate Excel results (right and wrong). Feel free to contribute to the effort by coding what you think is the right solution and I’ll add an ExcelCompliant flag to the function to conditionally invoke your code.

How do I use the library?
Just add Financial.dll to the references in your project. The functions are provided as static methods on a Financial class in the System.Numeric namespace
I see the library was implemented with F#. But I don’t want to redistribute F# along with my application. What should I do?
There are two versions of the library. One of them statically links the F# libraries so that there is no dependency on F#. However, this assembly larger, so if you have F# installed, you can use the FinancialNotStandalone.dll instead.

How do I run the tests?
Run FinancialTests.exe. You need Excel 12 for the tests to work because they use Excel to test that the results are correct. You don’t need Excel 12 to use the library in your own application.

How do I compile the library?
You need to have F# September CTP installed (you can get it from here). There are two batch files (CreateLibraryStandalone.bat and CreateLibraryNotStandalon.bat). Run them to compile the dll. You might have to change the path to the F# compiler inside these files

How do I compile the tests?
Run CreateTests.bat

Have you tested this thing?
Yes, I do have 201,349 testcases running against it. You can easily raise that number significantly by adding new values to test in testdef.fs. If you have a multiproc machine the testcases will run faster as I parallelize their execution.

Have you run performance tests on it?
Not at all. The only thing I checked is that all the recursive functions are tail recursive. Feel free to let me know if they are slow.

Are there any functions that behave different from Excel?
Yes, there are two of them.
CoupDays
The Excel algorithm seems wrong in that it doesn’t respect the following:
coupDays = coupDaysBS + coupDaysNC.
This equality should stand. By manually counting the days, I’m pretty confident that my algorithm is correct.My result differs from Excel by +/- one or two days when the date spans a leap year.
VDB
In the excel version of this algorithm the depreciation in the period (0,1) is not the same as the sum of the depreciations in periods (0,0.5) (0.5,1).
VDB(100,10,13,0,0.5,1,0) + VDB(100,10,13,0.5,1,1,0) <> VDB(100,10,13,0,1,1,0)
Notice that in Excel by using ‘1’ (no_switch) instead of ‘0’ as the last parameter everything works as expected. The last parameter should have no influence in the calculation given that in the first period there is no switch to sln depreciation.
Overall, I think my algorithm is correct, even if it disagrees with Excel when startperiod is fractional.

Can you list the functions with their testcases results?
Succeeded 1840/1840 for PV
Succeeded 2024/2024 for FV
Succeeded 2240/2240 for PMT
Succeeded 853/853 for NPER
Succeeded 5355/5355 for IPMT
Succeeded 5355/5355 for PPMT
Succeeded 208/208 for CUMIPMT
Succeeded 208/208 for CUMPRINC
Succeeded 624/624 for ISPMT
Succeeded 12/12 for FVSCHEDULE
Succeeded 9/9 for IRR
Succeeded 21/21 for NPV
Succeeded 147/147 for MIRR
Succeeded 18/18 for XIRR
Succeeded 396/396 for DB
Succeeded 24/24 for SLN
Succeeded 132/132 for SYD
Succeeded 456/456 for DDB
Succeeded 2544/2544 for VDB excluding fractional startdates
Succeeded 11520/11520 for AMORLINC
Succeeded 23040/23040 for AMORDEGRC
Succeeded 15/15 for COUPDAYS excluding leap years
Succeeded 915/915 for COUPDAYSBS
Succeeded 915/915 for COUPDAYSNC
Succeeded 915/915 for COUPNUM
Succeeded 915/915 for COUPPCD
Succeeded 915/915 for COUPNCD
Succeeded 360/360 for ACCRINTM
Succeeded 1920/1920 for ACCRINT
Succeeded 10980/10980 for PRICE
Succeeded 1940/1940 for PRICEMAT
Succeeded 2910/2910 for YIELDMAT
Succeeded 1395/1395 for YEARFRAC
Succeeded 2745/2745 for INTRATE
Succeeded 1290/1290 for RECEIVED
Succeeded 2745/2745 for DISC
Succeeded 3660/3660 for PRICEDISC
Succeeded 2745/2745 for YIELDDISC
Succeeded 48/48 for TBILLEQ
Succeeded 69/69 for TBILLYIELD
Succeeded 81/81 for TBILLPrice
Succeeded 12/12 for DOLLARDE
Succeeded 12/12 for DOLLARFR
Succeeded 12/12 for EFFECT
Succeeded 12/12 for NOMINAL
Succeeded 5490/5490 for DURATION
Succeeded 5490/5490 for MDURATION
Succeeded 19320/19320 for ODDFPRICE
Succeeded 30600/30600 for ODDLPRICE
Succeeded 45900/45900 for ODDLYIELD
Test Cases Succeeded 201349/201349

Bayesian inference in F# - Part IIa - A simple example - modeling Maia

Wed, 26 Nov 2008 15:41:00 GMT

Other parts:

Let’s start with a simple example: inferring the underlying attitude of a small baby by observing her actions. Let’s call this particular small baby Maia. People always asks her father if she is a ‘good’ baby or not. Her father started to wonder how he can possibly know that. Being ‘good’ is not very clear, so he chooses to answer the related question if her attitude is generally happy, unhappy or simply quiet (a kind of middle ground).

/// Underlying unobservable, but assumed stationary, state of the process (baby). Theta.
type Attitude =
    | Happy
    | UnHappy
    | Quiet

Her poor father doesn’t have much to go with. He can just observe what she does. He decides, for the sake of simplifying things, to categorize her state at each particular moment as smiling, crying or looking silly (a kind of middle ground).

/// Observable data. y.
type Action =
    | Smile
    | Cry
    | LookSilly

The father now has to decide what does it mean for Maia to be of an happy attitude. Lacking an universal definition of happiness in terms of these actions, he makes one up. Maia would be considered happy if she smiles 60% of the times, she cries 20% of the times and looks silly the remaining 20% of the times. He might as well have experimented with clearly happy/unhappy babies to come up with those numbers.

/// Data to model the underlying process (baby)
let happyActions = [ Smile, 0.6; Cry, 0.2; LookSilly, 0.2]
let unHappyActions = [Smile, 0.2; Cry, 0.6; LookSilly, 0.2]
let quietActions = [Smile, 0.4; Cry, 0.3; LookSilly, 0.3]

What does it mean exactly? Well, this father would call his wife at random times during the day and ask her if Maia is smiling, crying or looking silly. He would then keep track of the numbers and then somehow decide what her attitude is. The general idea is simple, the somehow part is not.

/// Generates a new uniformly distributed number between 0 and 1
let random =
    let rnd = new System.Random()
    rnd.NextDouble

We can now model Maia. We want our model to return a particular action depending on which attitude we assume Maia is in mostly. For example, if we assume she is an happy baby, we want our model to return Smile about 60% of the times. In essence, we want to model what happens when the (poor) father calls his (even poorer) wife. What would his wife tell him (assuming a particular attitude)? The general idea is expressed by the following:

/// Process (baby) modeling. How she acts if she is fundamentally happy, unhappy or quiet
let MaiaSampleDistribution attitude =
    match attitude with
    | Happy     -> pickOne happyActions
    | UnHappy   -> pickOne unHappyActions
    | Quiet     -> pickOne quietActions

The ‘pickOne’ function simply picks an action depending on the probability of it being picked. The name sample distribution is statistic-lingo to mean ‘what you observe’ and indeed you just can observe Maia’s actions, not her underlying attitude.

The implementation of pickOne gets technical. You don’t need to understand it to understand the rest of this post. This is the beauty of encapsulation. You can start reading from after the next code snippet if you want to.

‘pickOne’ works by constructing the inverse cumulative distribution function for the probability distribution described by the Happy/UnHappy/Quiet/Actions lists. There is an entry on wikipedia that describes how this works and I don’t wish to say more here except presenting the code.

/// Find the first value more or equal to a key in a seq<'a * 'b>.
/// The seq is assumed to be sorted
let findByKey key aSeq =
    aSeq |> Seq.find (fun (k, _) -> k >= key) |> snd
/// Simulate an inverse CDF given values and probabilities
let buildInvCdf valueProbs =
    let cdfValues =
        valueProbs
        |> Seq.scan (fun cd (_, p) -> cd + p) 0.
        |> Seq.skip 1
    let cdf =
        valueProbs
        |> Seq.map fst
        |> Seq.zip cdfValues
        |> Seq.cache
    fun x -> cdf |> findByKey x
/// Picks an 'a in a seq<'a * float> using float as the probability to pick a particular 'a
let pickOne probs =
    let rnd = random ()
    let picker = buildInvCdf probs
    picker rnd

Another way to describe Maia is more mathematically convenient and will be used in the rest of the post. This second model answers the question: what is the probability of observing an action assuming a particular attitude? The distribution of both actions and attitudes (observable variable and parameter) is called joint probability.

/// Another, mathematically more convenient, way to model the process (baby)
let MaiaJointProb attitude action =
    match attitude with
    | Happy     -> happyActions |> List.assoc action
    | UnHappy   -> unHappyActions |> List.assoc action
    | Quiet     -> quietActions |> List.assoc action

List.assoc returns the value associated with a key in a list containing (key, value) pairs. Notice that in general, if you are observing a process, you don’t know what its joint distribution is. But you can approximate it by running the MaiaSampleDistribution function on known babies many times and keeping track of the result. So, in theory, if you have a way to experiment with many babies with known attitudes, you can create such a joint distribution.

We now have modeled our problem, this is the creative part. From now on, it is just execution. We’ll get to that.

Expression tree serialization code posted on Code Gallery

Thu, 20 Nov 2008 15:39:20 GMT

Luke and I worked on this last year for one week doing pair programming. It is a good sample of how you can serialize LINQ expression trees to xml.

The sample includes these components:

An Expression Tree serialization API: A general purpose XML serialization of Expression Trees. This should work over any expression tree - though there are inevitably bugs. The serialization format is fairly crude, but has been expressive enough to support the variety of expression trees I’ve tried throwing at it.
A wrapper for serializing/deserializing LINQ to SQL queries: A wrapper around the expression serializer allows serializing LINQ to SQL queries and de-serializing into a query against a given DataContext.
A WCF service which accepts serialized query expression trees and executes against a back-end LINQ to SQL: To enable querying across tiers, a WCF service exposes service methods which execute serialized queries. The service implementation deserializes the queries against its LINQ to SQL connection.
An IQueryable implementation wrapping the client side of the WCF service: The client-side calling syntax is simplified by providing an IQueryable implementation. This implementation, RemoteTable, executes queries by serializing the query expression tree and calling the appropriate service. The object model that the service user is able to query against is imported by the WCF service reference per the DataContracts on the LINQ to SQL mapping on the server side

The sample is here. Enjoy!

Bayesian inference in F# - Part I - Background

Fri, 07 Nov 2008 11:48:00 GMT

Other posts:

My interest in Bayesian inference comes from my dissatisfaction with ‘classical’ statistics. Whenever I want to know something, for example the probability that an unknown parameter is between two values, ‘classical’ statistics seems to answer a different and more convoluted question.

Try asking someone what the 95% confidence interval for X is (x1, x2) means. Very likely he will tell you that it means that there is a 95% probability that X lies between x1 and x2. That is not the case in classical statistics. It is the case in Bayesian statistics. Also all the funny business of defining a Null hypothesis for the sake of proving its falseness always made my head spin. You don’t need any of that in Bayesian statistics. More recently, my discovery that statistical significance is an harmful concept, instead of the bedrock of knowledge I always thought it to be, shook my confidence in ‘classical’ statistics even more.

Admittedly, I’m not that smart. If I have an hard time getting an intuitive understanding of something, it tends to go away from my mind after a couple of days I’ve learned it. This happens all the time with ‘classical’ statistics. I feel like I have learned the thing ten times, because I continuously forget it. This doesn’t happen with Bayesian statistics. It just makes intuitive sense.

At this point you might be wandering what ‘classical’ statistics is. I use the term classical, but I really shouldn’t. Classical statistics is normally just called ‘statistics’ and it is all you learn if you pick up whatever book on the topic (for example the otherwise excellent Introduction to the Practice of Statistics). Bayesian statistics is just a footnote in such books. This is a shame.

Bayesian statistics provides a much clearer and elegant framework for understanding the process of inferring knowledge from data. The underlying question that it answers is: If I hold an opinion about something and I receive additional data on it, how should I rationally change my opinion?. This question of how to update your knowledge is at the very foundation of human learning and progress in general (for example the scientific method is based on it). We better be sure that the way we answer it is sound.

You might wander how it is possible to go against something that is so widely accepted and taught everywhere as ‘classical’ statistics is. Well, very many things that most people believe are wrong. I always like to cite old Ben on this: The fact that other people agree or disagree with you makes you neither right nor wrong. You will be right if your facts and your reasoning are correct.. This little rule always served me well.

In this series of posts I will give examples of Bayesian statistics in F#. I am not a statistician, which makes me part of the very dangerous category of ‘people who are not statisticians but talk about statistics. To try to mitigate the problem I enlisted the help of Ralf Herbrich, who is a statistician and can catch my most blatant errors. Obviously I’ll manage to hide my errors so cleverly that not even Ralf would spot them. In which case the fault is just mine.

In the next post we’ll look at some F# code to model the Bayesian inference process.

My F# PDC presentation is online (+ ppt and demo application)

Mon, 03 Nov 2008 11:33:00 GMT

You can watch it here. Also I attached the demo application I built on stage.

fsharpdemoPDCPrez.zip

Luca presenting F# at PDC 2008

Fri, 24 Oct 2008 12:21:14 GMT

Next Thursday at 1.45pm I will present “An introduction to Microsoft F#” at PDC 2008. If you are there, please stop by or drop by the lounge area to chat.

There are a bunch of other great sessions from my team. They are all good (I swear I tried to format John’s session correctly, but failed).

TL16: The Future of C#
Speaker: Anders Hejlsberg
Mon 10/27 | 1:45 PM-3:00 PM | Petree Hall CD
Repeated: Wed 10/29 | 3:00 PM-4:15 PM | 502A
TL10: Deep Dive: Dynamic Languages in .NET
Speaker: Jim Hugunin
Mon 10/27 | 3:30PM | 502A
TL11: An Introduction to Microsoft F#
Speaker: Luca Bolognese
Thu 10/30 | 1:45 PM-3:00 PM | 502A
TL12: Future Directions for Microsoft Visual Basic
Speaker: Paul Vick
Tue 10/28 | 5:15 PM-6:30 PM | 406A
TL54: Natural Interop with Silverlight and Office
Alex Turner
Tue 10/28 | 12:45 PM-1:30 PM | 404A
TL57: A Panel on the Future of Programming Languages
G. Bracha, D.Crockford, A. Hejlsberg, E. Meijer, W.Schulte, J. Siek
Wed 10/29 | 10:30 AM-11:45 AM | 403AB
TL44 IronRuby: The Right Language for the Right Job
l Speaker: John Lam
l Mon 10/27, 5:15PM, 515B

Downloading stock prices in F# - Part VI - Code posted

Mon, 20 Oct 2008 18:45:00 GMT

Other parts:

An unnamed friend told me that I should stop posting small snippets of code and instead post entire solutions on CodeGallery. I did it for this one and here is the link.

Here is what’s in the zip file:

BackTestF - main library to download stock prices
Common.fs - common things used in the rest of the project (i.e. data modeling and common funcs)
html.fs - functions to scrap html tables, rows and cells
loader.fs - this is where the main async downloading algorithms are implemented
persistence.fs - async saving and loading of stock prices to files
algorithms.fs - async calculations of compound yearly returns given tickers and dates
dotNetWrapper.fs - gives a .NET friendly interface to the whole library so that you can use it from C#/VB.NET
Tests - too few testcases running on xUnit (you need to download xUnit separately to run them
ReturnCalculator - simple console application to show usage of the library
Bob - rough winforms UI application that uses the library. An unnamed friend (Jonathan) promised me that he was going to create a UI for my little library. My requirements were very simple: I want the best UI app of this century, one that fully takes advantage of the async nature of my code. Bob doesn’t fully satisfy my requirements yet 🙂

The code in persistence.fs, algorithms.fs and especially dotnetwrapper.fs is pretty rough and uninteresting. This is why I don’t blog about it. I reserve the right to do it if I get around to clean it up.

Downloading stock prices in F# - Part V - Adjusting historical data

Fri, 26 Sep 2008 16:04:00 GMT

Other parts:

Here is the problem. When you download prices/divs/splits from Yahoo you get a strange mix of historical numbers and adjusted numbers. To be more precise, the dividends are historically adjusted. The prices are not adjusted, but there is one last column in the data for Adjusted close. If you don’t know what ‘adjusted’ means in this context read here.

The problem with using the ‘adjusted close’ column is that, for a particular date in the past, ‘adjusted close’ changes whenever the company pays a dividend or splits its stock. So if I retrieve the value on two different days I might get different numbers because, in the meantime, the company paid a dividend. This prevents me from storing a subset of the data locally and then retrieving other subsets later on. It also has the limitation that just the closing price is present while I might need adjusted opening price, adjusted high price or even adjusted volume depending on the operations I want to perform on the data (i.e. calculating oscillators or volume-adjusted moving averages).

The solution I came up with is to download the data and transform it to an ‘asHappened’ state. This state is simply an unadjusted version of what happened in the past. Data in this state is not going to change in the future, which means that I can safely store it locally. I can then on demand produce ‘historically adjusted’ data whenever I need to.

Ok, to the code. As it often happens, I need some auxiliary functions before I get to the core of the algorithms. The first one is a way to compare two observations, I will use it later on to sort a list of observations.

let compareObservations obs1 obs2 =
    if obs1.Date <> obs2.Date then obs2.Date.CompareTo(obs1.Date)
    else
        match obs1.Event, obs2.Event with
            | Price _, Price _ | Div _, Div _ | Split _, Split _
                -> failwith "Two same date/ same kind observations"
            | Price _, _  -> -1
            | _, Price _  -> 1
            | _           ->

This is rather simple. If the dates of these observations are different, just compare them. If they are the same then the two observations cannot be of the same type (i.e. I cannot have two prices for a particular date). Given that they are not of the same, then &(&^%!#$!4. Crap, that teaches me to put comments in my code! I think I’m putting the price information first, but I’m not sure. Anyhow my universal excuse not to figuring it out is that the testcases succeed so I must be doing it right (how lame, testcase-addiction I guess).

The next auxiliary function is just a wrapper over fold. I always tend to wrap fold calls in a method with a better name because I remember the old times when I didn’t know what fold was. I want a reader of my code to be able to understand it even if they are not familiar with fold (the ‘universal functional Swiss-Army-Knife). This function is a map that needs to know the value of an accumulator to correctly perform its mapping over each element.

let mapAcc acc newAccF newItemF inl =
    let foldF (acc, l) x = newAccF acc x, (newItemF acc x)::l
    let _, out = inl |> List.fold_left foldF (acc, [])
    out

Apart from the implementation details, this function takes an accumulator, an accumulator function, an item function and an input list. For each element in the list it calculates two things:

a new value for the accumulator: newAccumulatorValue = newAccF oldAccValue itemValue
a new value for the item: new ItemValue = newItemF accValue oldItemValue

Maybe there is a standard functional way to do such a thing with a specific name that I’m not aware of. Luke might know. He is my resident fold expert.

All right, now to he main algorithm.

let asHappened splitFactor observations =
    let newSplitFactor splitFactor obs =
        match obs.Event with
            | Split(factor) -> splitFactor * factor
            | _             -> splitFactor
    let newObs splitFactor obs =
        let date = obs.Date
        let event = match obs.Event with
                        | Price(p)                  -> Price(p)
                        | Div(amount)               -> Div(amount * splitFactor)
                        | Split(factor)             -> Split(factor)
        {Date = date; Event = event}
    observations
    |> List.sort compareObservations
    |> mapAcc splitFactor newSplitFactor newObs

To understand what’s going on start from the bottom. I’m taking the observation list downloaded from Yahoo and sorting it using my compareObservations function. I then take the resulting list and apply the previously described mapAcc to it. For this function splitFactor is the accumulator, newSplitFactor is the accumulator function and newObs is the function that generate a new value for each item in the list.

NewSplitFactor is trivial: every time it sees a Split observation it updates the value of the split factor. That’s it. NewObs is rather simple as well. Every time it sees a dividend, it ‘unadjust’ it by multiplying its amount by the split factor. The end result is to transform the dividends downloaded from Yahoo (which are adjusted) to an unadjusted state. I could have filtered out the price observations before doing all of this and add them back afterward, but didn’t. It’d probably be slower

Now that I can recreate the state of the world as it was at a particular point in time, what if I want to adjust the data? I can call adjusted

let adjusted (splitFactor, lastDiv, oFact, hFact, lFact, cFact, vFact)
                                                                   asHappenedObs =
    let newFactor (splitFactor, lastDiv, oFact, hFact, lFact, cFact, vFact) obs =
        match obs.Event with
            | Split(split)  ->
                 splitFactor * split, lastDiv, oFact, hFact, lFact, cFact, vFact
            | Div(div)      -> splitFactor, div, oFact, hFact, lFact, cFact, vFact
            | Price(p)      ->
                 splitFactor, 0.<money>, oFact / (1. - lastDiv / p.Open),
                 hFact / (1. - lastDiv / p.High), lFact / (1. - lastDiv / p.Low),
                 cFact / (1. - lastDiv / p.Close), vFact / (1. - lastDiv / p.Close)
    let newObs (splitFactor, lastDiv, oFact, hFact, lFact, cFact, vFact) obs =
        let date = obs.Date
        let event = match obs.Event with
                        | Price(p)          ->
                            Price({Open = p.Open / splitFactor / oFact;
                            High = p.High / splitFactor / hFact;
                            Low = p.Low / splitFactor / lFact;
                            Close = p.Close / splitFactor / cFact;
                            Volume = p.Volume / splitFactor / vFact })
                        | Div(amount)       -> Div (amount / splitFactor)
                        | Split(split)      -> Split(split)
        {Date = date; Event = event}
    asHappenedObs
        |> List.sort compareObservations
        |> mapAcc (splitFactor, lastDiv, oFact, hFact, lFact, cFact, vFact)
                                                                  newFactor newObs
        |> List.filter (fun x -> match x.Event with Split(_) -> false | _ -> true)

Wow, ok, this looks messy. Let’s go through it. Starting from the bottom: sort the observations, perform the right algorithm and filter away all the splits. It doesn’t make sense to have splits in adjusted data.

The interesting piece is the mappAcc function. It take a tuple of factors as accumulator and the usual two functions to update such tuple and create new observations. The newObs function creates a new Observation using the factors in the accumulator tuple. Notice how the dividends are divided by the splitFactor (which is the opposite of our asHappened algorithm where we were multiplying them). Also notice how the prices are divided by both the splitFactor and the pertinent price factor. This is needed because the prices need to be adjusted by the dividends paid out and the adjustment factor is different for each kind of price (i.e. open, close, etc). The newFactor function simply updates all the factors depending on the current observation.

Notice how asHappened and adjusted are structurally similar. This is an artifact of having a functional approach to writing code: it kind of forces you to identify these commonality in the way an algorithm behave and abstract them out (in this case in the mapAcc function). You often discover that such abstracted-out pieces are more generally useful than the case at hand.

Downloading stock prices in F# - Part IV - Async loader for splits

Fri, 19 Sep 2008 17:59:00 GMT

Other parts:

Downloading splits is a messy affair. The problem is that Yahoo doesn’t give you a nice comma-delimitated stream to work with. You have to parse the Html yourself (and it can be on multiple pages). At the end of the post, the overall result is kind of neat, but to get there we need a lot of busywork.

First, let’s define a function that constructs the correct URL to download splits from. Notice that you need to pass a page number to it.

let splitUrl ticker span page =
    "http://finance.yahoo.com/q/hp?s=" + ticker + "&a="
    + (span.Start.Month - 1).ToString() + "&b=" + span.Start.Day.ToString() + "&c=" 
    + span.Start.Year.ToString() + "&d=" + (span.End.Month - 1).ToString() + "&e="
    + span.End.Day.ToString() + "&f=" + span.End.Year.ToString() + "&g=v&z=66&y="
    + (66 * page).ToString();

The reason for this particular url format (i.e. 66 * page) is completely unknown to me. I also have the feeling that it might change in the future. Or maybe not given how many people rely on it.

I then describe the driver function for loading splits:

let rec loadWebSplitAsync ticker span page splits =
    let parseSplit text splits =
        List.append splits (parseSplits (scrapHtmlRows text)),
                                           not(containsDivsOrSplits (scrapHtmlCells text))
    async {
        let url = splitUrl ticker span page
        let! text = loadWebStringAsync url
        let splits, beyondLastPage = parseSplit text splits
        if beyondLastPage then return splits else
                                 return! loadWebSplitAsync ticker span (page + 1) splits }

This is a bit convoluted (it is an Async recursive function). Let’s go through it in some detail. First there is a nested function parseSplit. It takes an html string and a list of observations and returns a tuple of two elements. The first element is the same list of observations augmented with the splits found in the text. The second element is a boolean that is true if we have navigated beyond the last page for the splits.

The function to test that we are beyond the last page is the following:

let containsDivsOrSplits cells =
    cells |> Seq.exists
        (fun (x:string) -> Regex.IsMatch(x, @"$.+Dividend", RegexOptions.Multiline)
                           || Regex.IsMatch(x, "Stock Split"))

This function just checks if the words Stock Split or Dividend are anywhere in the table. If they aren’t, then we have finished processing the pages for this particular ticker and date span.

The function to extract the splits observations from the web page takes some cells (a seq>) as input and returns an observation list. It is reproduced below:

let parseSplits rows =
    let parseRow row =
        if row |> Seq.exists (fun (x:string) -> x.Contains("Stock Split"))
        then
            let dateS = Seq.hd row
            let splitS = Seq.nth 1 row
            let date = DateTime.Parse(dateS)
            let regex = Regex.Match(splitS,@"(d+)s+:s+(d+)s+Stock Split",
                                                                   RegexOptions.Multiline)
            let newShares = shares (float (regex.Groups.Item(1).Value))
            let oldShares = shares (float (regex.Groups.Item(2).Value))
            Some({Date = date; Event = Split(newShares / oldShares)})
        else None
    rows |> Seq.choose parseRow |> Seq.to_list

It just take a bunch of rows and choose the ones that contain stock split information. For these, it parses the information out of the text and creates a Split Observation out of it. I think it is intuitive what the various Seq functions do in this case. Also note my overall addiction to the pipe operator ( |> ). In my opinion this is the third most important keyword in F# (after ‘let’ and ‘match’).

Let’s now go back to the loadWebSplitAsync function and discuss the rest of it. In particular this part:

async {
    let url = splitUrl ticker span page
    let! text = loadWebStringAsync url
    let splits, beyondLastPage = parseSplit text splits
    if beyondLastPage then return splits else
        return! loadWebSplitAsync ticker span (page + 1) splits }

First of all it is an Async function. You should expect some Async stuff to go on inside it. And indeed, after forming the URL in the first line, the very next line is a call to loadWebStringAsync. We discussed this one in the previous installment. It just asynchronously loads a string from an URL. Notice the bang after ‘let’. This is your giveaway that async stuff is being performed.

The result of the async request is parsed to extract splits. Also, the beyondLastPage flag is set if we have finished our work. If we have, we return the split observation list; if we haven’t, we do it again incrementing the page number to load the html text from.

Now that we have all the pieces in places, we can wrap up the split loading stuff inside this facade function:

let loadSplitsAsync ticker span = loadWebSplitAsync ticker span 0 []

And finally put together the results of this post and the previous one with the overall function-to-rule-them-all:

let loadTickerAsync ticker span =
    async {
        let prices = loadPricesAsync ticker span
        let divs =  loadDivsAsync ticker span
        let splits = loadSplitsAsync ticker span
        let! prices, divs, splits = Async.Parallel3 (prices, divs, splits)
        return prices |> List.append divs |> List.append splits
        }

All right, that was a lot of work to get to this simple thing. This is a good entry point to our price/divs/split loading framework. It has the right inputs and outputs: it takes a ticker and a date span and returns an Async of a list of observations. Our caller can decide when he wants to execute the returned Async object.

Notice that in the body of the function I call Async.Parallel. This is debatable. A more flexible solution is to return a tuple containing three Asyncs (prices, divs, splits) and let the caller decide how to put them together. I decided against this for simplicity reasons. This kind of trade-off is very common in Async programming: giving maximum flexibility to your caller against exposing something more understandable.

I have to admit I didn’t enjoy much writing (and describing) all this boilerplate code. I’m sure it can be written in a better way. I might rewrite plenty of it if I discover bugs. I kind of like the end result though. loadTickerAsync has an overall structure I’m pretty happy with.

Next post, some algorithms with our observations

Downloading stock prices in F# - Part III - Async loader for prices and divs

Fri, 12 Sep 2008 16:18:00 GMT

Other parts:

It is now time to load our data. There is a bit of uninteresting code to start with, but things get interesting afterward. Let’s start with functions that create the right URLs to download prices and dividends. We’ll talk about splits in the next installment.

let commonUrl ticker span =
    @"http://ichart.finance.yahoo.com/table.csv?s=" + ticker + "&a="
    + (span.Start.Month - 1).ToString() + "&b=" + span.Start.Day.ToString() + "&c="
    + span.Start.Year.ToString() + "&d=" + (span.End.Month - 1).ToString() + "&e=" 
    + span.End.Day.ToString() + "&f=" + span.End.Year.ToString()
let priceUrl ticker span = commonUrl ticker span + "&g=d&ignore=.csv"
let divUrl ticker span = commonUrl ticker span + "&g=v&ignore=.csv"

We will also need to construct an observation given a comma delimitated line of text. Again, for spits things will be harder.

let parsePrice (line: string) =
    let tokens = line.Split([|','|])
    { Date = DateTime.Parse(tokens.[0]);
      Event = Price ({Open = money (Double.Parse(tokens.[1])) ;
      High = money (Double.Parse(tokens.[2]));
      Low = money (Double.Parse(tokens.[3])); Close = money (Double.Parse(tokens.[4]));
      Volume = volume (Double.Parse(tokens.[5]))})}
let parseDiv (line: string) =
    let tokens = line.Split([|','|])
    let date = DateTime.Parse(tokens.[0])
    let amount = money (Double.Parse(tokens.[1]))
    {Date = date; Event = Div amount}

Nothing noteworthy about this code. We have a couple of other ‘infrastructure pieces before we get to the Async pieces. The next function is recursive. It takes a StringReader and reads lines out of it. For each line it calls a parsing function that takes the line as input and returns an object as output. The function gathers all such objects in the listOfThings list. If you are new to F# the following construct (parseLineFunc line:: listOfThings) means: execute the parseLineFunc with argument line, take the result and create a list that has the result as head and listOfThings as tail).

let rec loadFromLineReader (reader:StringReader) listOfThings parseLineFunc =
    match  reader.ReadLine () with
    | null  -> listOfThings
    | line  -> loadFromLineReader reader (parseLineFunc line::listOfThings) parseLineFunc

The next function is rather uninteresting. It just converts a string to a StringReader, cut out the first line (header) and calls loadFromLineReader.

let loadFromLineString text listOfThings parseLineFunc =
    let reader = new StringReader(text)
    reader.ReadLine ()|> ignore // skip header
    loadFromLineReader reader listOfThings parseLineFunc

We now come to the first Async function. But what is an Async function? There are several possible technically correct definition as: it is an instance of the monad pattern or it is a function that returns an Async object or it is a way to release your thread to the thread pool. These definition don’t help me much. I need something intuitive to latch one.

The way that I personally visualize it is: there are things in the world that are very good at executing certain tasks and like to be hit by multiple parallel requests for these tasks. They’d like me to give them their workload and get out of their way. They’ll call me when they are done with it. These ‘things’ are disk drives, web servers, processors, etc Async is a way to say: hey, go and do this, call me when you are done.

Now, you can call the asynchronous APIs directly, or you can use the nice F# language structures to do it. Let’s do the latter.

let loadWebStringAsync url =
    async {
        let req = WebRequest.Create(url: string)
        use! response = req.AsyncGetResponse()
        use reader = new StreamReader(response.GetResponseStream())
        return! reader.AsyncReadToEnd()}

This function retrieves a web page as a string asynchronously. Notice that even if the code looks rather normal, this function will likely be executed on three different thread. The first thread is the one the caller of the function lives on. The function AsyncGetResponse causes the thread to be returned to the thread pool waiting for a response back from the web server. Once such a response arrives, the execution resumes on a different thread until AsyncReadToEnd. That instruction returns the execution thread to the thread pool. A new thread is then instantiated when the string has been completely read. The good thing is that all of this is not explicitly managed by the programmer. The compiler ‘writes the code’ to make it all happen. You just have to follow a set of simple conventions (i.e. putting exclamation marks in the right place).

The return result of this function is an Async, which is something that, when executed, returns a string. I cannot emphasize this enough: always look at the signature of your F# functions. Type inference can be tricky

Async is somehow contagious. If you are calling an Async function you have to decide if propagate the Asyncness to your callers or remove it by executing the function. Often propagating it is the right thing to do as your callers might want to batch your function with other aync ones to be executed together in parallel. Your callers have more information than you do and you don’t want to short-circuit them. The following function propagates ayncness.

let loadFromUrlAsync url parseFunc =
    async {
        let! text = loadWebStringAsync url
        return loadFromLineString text [] parseFunc}

Let’s see how the functions presented to this point compose to provide a way to load prices and dividends (splits will be shown afterward).

let loadPricesAsync ticker span = loadFromUrlAsync (priceUrl ticker span) parsePrice
let loadDivsAsync ticker span = loadFromUrlAsync (divUrl ticker span) parseDiv

This composition of functions is very common in functional code. You construct your building blocks and assemble them to achieve your final goal. Functional programming is good at almost forcing you to identify the primitive blocks in your code. All right, next in line is how to load splits.

Downloading stock prices in F# - Part II - Html scraping

Fri, 05 Sep 2008 14:41:00 GMT

Other parts:

Getting stock prices and dividends is relatively easy given that, on Yahoo, you can get the info as a CVS file. Getting the splits info is harder. You would think that Yahoo would put that info in the dividends CVS as it does when it displays it on screen, but it doesn’t. So I had to write code to scrap it from the multiple web pages where it might reside. In essence, I’m scraping this.

html.fs

In this file there are utility functions that I will use later on to retrieve split info.

#light
open System
open System.IO
open System.Text.RegularExpressions

// It assumes no table inside table ...
let tableExpr = "<table[^>]*>(.*?)</table>"
let headerExpr = "<th[^>]*>(.*?)</th>"
let rowExpr = "<tr[^>]*>(.*?)</tr>"
let colExpr = "<td[^>]*>(.*?)</td>"
let regexOptions = RegexOptions.Multiline ||| RegexOptions.Singleline 
                                          ||| RegexOptions.IgnoreCase

This code is straightforward enough (if you know what Regex does). I’m sure that there are better expression to scrap tables and rows on the web, but these work in my case. I really don’t need to scrape tables. I put the table expression there in case you need it.

I then write code to scrape all the cells in a piece of html:

let scrapHtmlCells html =
  seq { for x in Regex.Matches(html, colExpr, regexOptions) -> x.Groups.Item(1).ToString()}

This is a sequence expression. Sequence expressions are used to generate sequences starting from some expression (as the name hints to). In this case Regex.Matches returns a MatchClollection, which is a non-generic IEnumerable. For each element in it, we return the value of the first match. We could as easily have constructed a list or an array, given that there is not much deferred computation going on. But oh well

Always check the type of your functions in F#! With type inference it is easy to get it wrong. Hovering your mouse on top of it in VS shows it. This one is typed: string -> seq. It takes a string (html) and return a sequence of strings (the cells in html).

We’ll need rows as well.

let scrapHtmlRows html =
    seq { for x in Regex.Matches(html, rowExpr, regexOptions) -> scrapHtmlCells x.Value }

This works about the same. I’m matching all the rows and retrieving the cells for each one of them. I’m getting back a matrix-like structure, that is to say that this function as type: string -> seq>.

That’s all for today. In the next installment we’ll make it happen.

Downloading stock prices in F# - Part I - Data modeling

Fri, 29 Aug 2008 19:13:00 GMT

Other parts:

Today we shipped the September CTP of F# !!!! Evviva !! Read this blog post about it. To celebrate I decided to share one of my several F# project. It might make for a good sample; sort of a crash course on F#.

This application downloads stock prices, dividends and splits from Yahoo Historical Prices and performs computations on them. I will describe it file by file.

common.fs

I always have such a file in my projects. It is a repository for the types that I’m going to use in my program and the functions that are common across multiple modules. If I have a large program I also have a types.fs just for the types.

This one starts like this:

#light
open System
[<Measure>] type money
let money (f:float) = f * 1.<money>
[<Measure>] type shares
let shares (f:float) = f * 1.<shares>
[<Measure>] type volume
let volume (f:float) = f * 1.<volume>
[<Measure>] type rateOfReturn
let rateOfReturn (f:float) = f * 1.<rateOfReturn>

The first line instructs the compiler to use the lightweight syntax. You don’t want to know what the heavyweight syntax is. Just always put such a line at the start of your files. The next line opens up the System namespace. Then the good stuff starts.

I’m defining some units of measures. The simplest way to think about units of measures is: they are a type system for floats. You can do much more than that, but it is a good first approximation. For example, you cannot now sum a money type and a volume type. Also for each one I define a function that converts from a normal float type to it (if you come from a C# background, floats are doubles).

Then I define the data model for my application:

type Span = { Start: DateTime; End: DateTime }
type Price = { Open: float<money>; High: float<money>; Low:float<money>; 
                                                Close:float<money>; Volume: float<volume>}
type Event =
    | Price of Price
    | Split of float
    | Div of float<money>
type Observation = { Date: DateTime; Event: Event}

The first record that I define, Span, represents the difference between two dates. It is just a little useful thing. A more fundamental record is Observation. An Observation is defined as something that happens on a particular Date. That something, an Event, can be one of three things: a Price, a Split or a Div. A Price is another record with a bunch of float fields and on float field. If you go to the Yahoo site, you’ll see what it represents.

A Split is simply a float. Why not a float<>? Because it is just a number, a factor to be precise. It represents the number of new shares divided by the number of old shares. float / float = float. A Div is a float.

This is one way to model the problem. Infinite other ways are possible (and I tried many of them in a C# version of this code that ended up using polymorphism). Note that all of the types are records except Event that is a discriminated union.

Records are read only containers of data. Discriminated unions are what the name says: things that can be one of multiple things (even recursively). They are rather handy to represent the structure of the data. We will see how you use them using the match operator in upcoming posts.

Also notice the following common pattern in F# (and functional programming in general). You define your data and then you define transformations over it. F# has a third optional step, that is to expose these transformations as methods of a .NET objects.

We are almost done here. A handful of other functions are in my file:

let span sy sm sd ey em ed =
                   {Start = new DateTime(sy, sm, sd); End = new DateTime(ey, em, ed)}
let date y m d = new DateTime(y, m, d)
let now () = DateTime.Now
let idem x = x
let someIdem x = Some(x)

Span is a function that creates a span given the relevant info. date creates a date given year, month and day. now is a value that corresponds to the current date. idem is a function that returns its parameter (you’ll see how that can possibly be useful). someIdem is a function that unpack an Option type and gives his value. I could write all my code without these things, but it looks better (to me) with them.

Notice that in the F# code you have access to all the functions and type in the .NET framework. It is just another .NET language: it can create or consume .NET types.

All right, we are done for part one. In part II there will be some real code.

A C# library to write functional code - Part V - The Match operator

Tue, 15 Jul 2008 05:46:00 GMT

Other posts in the series:

This is my last post of this series. It is about the match operator. To the untrained eyes this operator might look like a case statement. But they are different.

The match operator combines control flow and decomposition in a single construct. As it often happens, even if these two things are well known concepts, putting them together gives you a different perspective on your code.

I have to admit that the similarity with a case statement triggered all sorts of bad reactions in my OO trained mind. Phrases from wise gurus on the tone of never use an ‘if’ statement was echoing in my mind. After a while I got over it and enjoyed the power that this operator gives.

In this library we won’t get close to the beauty of the match operator in functional languages (i.e. F#), especially the decomposition piece. There might be a way to do better than this in C#, but this was enough for my purpose of learning about functional programming, so I didn’t investigate further. Learning was my real goal.

How to use it

There are two versions of this operator. Let’s start from the more eye pleasing one.

Let’s assume a discriminated union like the following:

public class Node : TypeUnion<int, string> {
    public Node(int i) : base(i) { }
    public Node(string s) : base(s) { }
    public int IntNode { get { return Type1; } }
    public string StringNode { get { return Type2; } }
}

I can then match against this union with the following code:

var no = new Node(35);
r = F.Match(no,
                (int i)    => (i + 3).ToString(),
                (string s) => s
            );
Assert.AreEqual("38", r);

Note that the match operator behaves a bit like a case statement, but it also gives you the ‘right’ type on the right of the ‘=>’ for you to write code against.

I have to admit I’m rather happy of this syntax, but it has one severe limitation. You need to specify all the types of the type union and they have to be in the same order. For example, in the previous code I cannot match against string first and int second. I consider this to be a big deal.

A different implementation of match that doesn’t have that limitation is as follows:

no = new Node("35");
r = F.Match(no,
    n => n.Is<int>(),   n => (n.As<int>() + 3).ToString(),
    n => n.Is<string>(),n => n.As<string>());
Assert.AreEqual("35", r);

Rather less pleasing to the eyes, but more robust to use.

By using this more generic version, you can obviously match against all the constructs we described in this series. I.E. against Tuples:

var t = F.Tuple("msft", 10, "Nasdaq");
var r = F.Match(t,
    i => i.Item2 < 5 && i.Item3 == "OTC",          i => i.Item1 + " is low price OTC stock",
    i => i.Item3 == "OTC",                         i => i.Item1 + "is a normal OTC stock",
    i => i.Item3 == "Nasdaq",                      i => i.Item1 + " is a Nasdaq stock");
Assert.AreEqual("msft is a Nasdaq stock", r);

Or against sequences:

var i1 = new int[] { 1, 2, 3, 4, 5, 6, 7 };
var r1 = F.Match(i1,
                s => s.SequenceEqual(new int[] { 1, 2}),    s => s.Where(i => i < 4),
                s => s.First() == 2,                        s => s.Where(i => i == 1),
                s => s.Last() == 6,                         s => s.Select(i => i * i),
                s => true,                                  s => s.Where(i => i < 7));
Assert.IsTrue(i1.Take(6).SequenceEqual(r1));

The match operator, as I defined it, is very flexible (probably too flexible as it doesn’t use the type system to enforce much).

How it is implemented

Let’s start from the special match against union types: the one that is beautiful but flawed. Its implementation looks just like this (I just show the two parameters version):

public static R Match<T1, T2, R>(
                           this TypeUnion<T1, T2> u,
                           Func<T1, R> f1, Func<T2, R> f2) {
    if (u.Is<T1>()) return f1(u.As<T1>());
    if (u.Is<T2>()) return f2(u.As<T2>());
    throw new Exception("No Match for this Union Type");
}

It is easy to see the reason for the limitations of this function. The same thing that gives you type inference (nice to look at) also gives you the problem with the ordering of lambdas. Also note the I originally intended to have these as extension methods. In practice I ended up liking more the F.Match syntax. De gustibus I assume

The more general version looks like this:

public static R Match<T, R>(this T t,
    Func<T, bool> match1, Func<T, R> func1,
    Func<T, bool> match2, Func<T, R> func2) {
    if (match1(t))
        return func1(t);
    if (match2(t))
        return func2(t);
    throw new Exception("Nothing matches");
}

Again, it is easy to see in the implementation that this is an extremely general thing: a glorified case statement really. This is the beauty of it (you can match against anything) and the ugliness of it (you can write anything as match1 of func1, even code that doesn’t reference t at all).

How to use this series to learn functional programming

Here is how I did it (it worked for me). Take this library and force yourself to write a medium size program using just these constructs.

I mean it. No objects, just records. No inheritance, just discriminated unions. Tuples to return values. Match operators everywhere. No iteration statements (for, while, etc), just recursion. Extensive use of sequence operators. After a while I noticed that all my functions had the same pattern: they just match against the input and produce an output, usually by calling other functions or recursively calling themselves.

Obviously, this is overcompensating. After a while you will realize the different trade offs that come with a functional vs OO style of programming. At that point you’ll be able to get the best of the two. Or maybe you’ll be royally confused

At that point I picked up F# and I’m now enjoying the fact that all these constructs are directly embedded in the language (and yes, you can use while statements too).

A C# library to write functional code - Part IV - Type Unions

Fri, 06 Jun 2008 04:45:00 GMT

Other posts in the series:

I’m sorry for my prolonged absence in the middle of this series of posts. I’m on a long paternity leave in Italy (playing beach volley every day). It’s hard to have the discipline

A bunch of you wrote telling me to finish this. So here I go: let’s talk about type unions. First of all: they are not called like that. The correct name is discriminated unions. I have no idea why I call them differently, but I want to be consistent with my previous mistake.

For those of you with a C++ background (like myself) they are like unions, just better (or worse depending on your convictions). They let you define a type that can represent one of several different types. You can then use the ‘match’ operator (discussed in the next post) to pattern match against it.

I won’t elaborate on the pros and cons of this style of programming versus using polymorphism. I just want to show you how I implemented this construct in C#. As always, my usual caveat: this is just ‘educational code’, use it at your own risk, no extensive or perf related test has been done on it. You can download the zip file and check my unit tests for yourself.

How type unions are used

In my world, you declare a type union like this:

public class Person { }
public class Dog { }
public class Friend : TypeUnion<Person, Dog> {
    public Friend(Person p) : base(p) { }
    public Friend(Dog d) : base(d) { }
}

You inherit a type union from the TypeUnion class and use generic parameters that correspond to the types that the union can represent.

You can then create a type union as:

var fr = new Friend(new Dog());

Test its type by:

Assert.IsTrue(fr.Is<Dog>());
Assert.IsFalse(fr.Is<Person>());

Cast it to one of the types they represent:

var d = fr.As<Dog>();

Or use it with the ‘match’ operator (fully explained in an upcoming post):

var r = F.Match(fr,
    f => f.Is<Dog>(), f => f.As<Dog>().ToString(),
    f => f.Is<Person>(), f => f.As<Person>().ToString());
Assert.AreEqual(r, new Dog().ToString());

Or the slightly more pleasing:

r = F.Match(fr,
        (Person p) => p.ToString(),
        (Dog d) => d.ToString());
Assert.AreEqual(r, new Dog().ToString());

You get the idea.

How they are implemented

Nothing really sophisticated going on here. Let’s take as an example a type union that can represent two types. I have versions that go to 5 types in the zip file.

First of all a TypeUnion is a Record:

public class TypeUnion<T1, T2> : Record<T1, T2> {

It has overloaded constructors to create a type union of a particular type:

public TypeUnion(T1 t1)
    : base(t1, default(T2)) {
    UnionType = t1.GetType();
}
public TypeUnion(T2 t2)
    : base(default(T1), t2) {
    UnionType = t2.GetType();
}

UnionType is used to ‘remember’ which type it is:

protected Type UnionType;

It also has properties to return the objects of all the types that can be stored:

protected T1 Type1 { get { return state.Item1; } }
protected T2 Type2 { get { return state.Item2; } }

The ‘Is’ operator is simply implemented as:

public bool Is<K>() {
    return typeof(K).IsAssignableFrom(UnionType);
}

And the ‘As’ operator looks like so:

public K As<K>() {
    if (!Is<K>())
        throw new Exception(string.Format(
          "In a TypeUnion cannot cast from {0} to {1}",
          UnionType.Name, typeof(K).Name));
    if (typeof(T1) == UnionType)
        return (K)(object) Type1;
    if (typeof(T2) == UnionType)
        return (K)(object) Type2;
    throw new Exception("Shouldn't get here");
}

I leave as an exercise to the reader to understand what happens if T1 and T2 are the same type or inherit from the same type. I could have written code to handle this case in a more explicit manner, but didn’t.

Also, by reviewing my code I found an obvious bug in my Is/As code. I fixed it and re-posted the zip file in the second post of this series.

Now back to the beach. Next post is on the ‘match’ operator.

A C# library to write functional code - Part III - Records

Mon, 21 Apr 2008 13:34:00 GMT

Other posts in the series:

Now that we know what Tuples are, we can start talking about Record, as they use a derivative of Tuples under the cover. But first, what is a record?

Well, in C# parlance a Record is a sort of immutable value object. I talked at length about these fellows in this series of blog posts. In functional parlance, a Record is a Tuple that lets you access its items by name.

You code Records like this:

public class Order: Record<string, int> {
    public Order(string item, int qty): base(item,qty) {}
    public string Item { get { return state.Item1;}}
    public int Quantity { get { return state.Item2; } }
}
public class Customer: Record<string, IEnumerable<Order>> {
    public Customer(string name, IEnumerable<Order> orders) : base(name, orders) { }
    public string Name { get { return state.Item1; } }
    public IEnumerable<Order> Orders { get { return state.Item2; } }
}

You need to do three things:

Inherit from a generic Record class specifying the types of the properties as parameters
Add a constructor that calls back to the Record constructor
Add getters to retrieve the values of the properties (no setters as it is immutable)

This may seem like plenty of work, but in return you get structural equality. Coding that by hand every time you use a record would be a royal pain. You lose control of the base class, but that is often not a problem as in functional-land you more often use type unions than inheritance.

How is it implemented?

First of all it is an abstract class with as many generic parameters as properties that you need in your Record. Let’s use two as an example.

public abstract class Record<T1, T2> {

This abstract class has a field of type STuple:

protected readonly STuple<T1, T2> state;

What is a STuple? Well it is exactly the same as the Tuple described in Part II, but coded as a struct instead of a class. The reason to use a struct is to not allocate an additional object on the stack. This allows this solution to be as ‘performant’ as simply having coded the fields on the class itself. Or at least I think so

The Record class also has a constructor that simply initialize the STuple:

public Record(T1 t1, T2 t2) { state = F.STuple(t1, t2); }

where

internal static STuple<T1, T2> STuple<T1, T2>(T1 t1, T2 t2) {
    return new STuple<T1, T2>(t1, t2);
}

The Equals method is very much the same as the Tuple’s one, just delegating to the same DSEqual function that checks equality for Tuples.

public override bool Equals(object right) {
    Utils.CheckNull(right);
    if (object.ReferenceEquals(this, right))
        return true;
    if (this.GetType() != right.GetType())
        return false;
    var rightT = right as Record<T1, T2>;
    return F.DSEquals(this.state, rightT.state);
}

That’s it. Not too difficult as most of the implementation is based on the Tuple’s code. Next post will hopefully be more interesting. It is about Type Unions.

A C# library to write functional code - Part II - Tuples

Tue, 08 Apr 2008 16:51:00 GMT

Other posts in the series:

Tuples are a way for you not to name things. In Object Oriented languages you got to name everything. If you need to represent a bunch of data, you create a class for it.

There is a strange asymmetry in mainstream OO languages in that you can pass multiple parameters to a function, but you can return just one value. Granted, there are ways around it: you can use ‘ref’ in C# or return some sort of collection where things are stored. But by and large the model is: you pass many, you get one; if you need to return more than one, create a class to represent this ‘bunch of data’. Tuples are a way for you not to create such a class.

Tuples are also much more than that. Once you have the language concept of ‘a bunch of data without a name’, you can create arrays of them, you can pass them as parameters, use them as local variables. Wherever you’d use a type, you can use a Tuple instead.

This is particularly appealing to me as I like to use classes almost exclusively to represent things that have a counterpart in the domain I’m modeling (i.e. Customer, Account). I don’t like to create classes/structs just for the sake of temporarily put some data together.

You can create your own Tuple class in C#, but the syntax gets ugly. Syntax matter. Syntax helps you to think differently about your program. We have syntax for anonymous types, but given that they cannot escape the scope of a method, they cannot be used as full replacement for Tuples.

In any case, to my implementation. Here is how you create a Tuple:

var t1 = F.Tuple(34, "bo", 2.3);

not too bad. In F# it is better: (34, bo, 2.3). And you often don’t need the parenthesis. But still, my C# version is ok.

You then need to access its elements:

var n = t1.Item1;
var s = t1.Item2;

In F# you usually access them by doing pattern matching, which gives a more intuitive syntax. But again, my C# syntax is not terrible.

Tuples need to have structural equality, which means that the following has to work:

ArrayList mad1 = new ArrayList { new List<IEnumerable<string>> { new string[] { "bo" }, new string[] { "bo" } },
                               32, "bo", new int[] { 4, 5, 6 } };
ArrayList mad2 = new ArrayList { new List<IEnumerable<string>> { new string[] { "bo" }, new string[] { "bo" } },
                               32, "bo", new int[] { 4, 5, 6 } };
ArrayList mad3 = new ArrayList { new List<IEnumerable<string>> { new string[] { "bo" }, new string[] { "bo" } },
                               32, "bo", new int[] { 4, 5, 5 } };

Assert.AreEqual(F.Tuple(mad1, mad2, mad1), F.Tuple(mad2, mad1, mad2));
Assert.AreNotEqual(F.Tuple(mad1, mad2, mad1), F.Tuple(mad1, mad3, mad1));

You can use Tuples as return values, parameters, locals etc. Unfortunately, the syntax is ugly when Tuples are part of the signature of a function:

public Tuple<string, IEnumerable<Tuple<string, ObservationHistory>>> Execute() {
}

With the above information, you can be a user of Tuples. From this point on, I’ll talk about some details of the implementation (I also attach the full code to this post as a zip file).

public class Tuple<T1> {
    public Tuple(T1 t1) {
        Item1 = t1;
    }
    public readonly T1 Item1;
    #region Equals, GetHashCode, ==, !=
}
public class Tuple<T1, T2> : Tuple<T1> {
    public Tuple(T1 t1, T2 t2) : base(t1) { Item2 = t2; }
    public readonly T2 Item2;
    #region Equals, GetHashCode, ==, !=
}

So, Tuples are classes, not structs. The reason for it is fully described in this series of posts. They also inherit from one another. There are pros and cons to that. The main pros are that I had to write less code and that you can pass a Tuple when a function expects a Tuple. The main drawback is that you can pass a Tuple when a function expects a Tuple. Also notice the use of public fields. These is a problem with frameworks that insist on properties (i.e. Data Binding). Also, I just got to 5 as arity goes. The day I need 6 items, I’ll add another one. It is boilerplate code (that I’d still like not to write).

The Equals method is a bit convoluted:

internal static class Utils {
    public static void CheckNull<T>(T t) {
        if (t == null)
            throw new ArgumentNullException();
    }
}

public override bool Equals(object right) {
    Utils.CheckNull(right);
    if (object.ReferenceEquals(this, right))
        return true;
    if (this.GetType() != right.GetType())
        return false;
    var rightT = right as Tuple<T1, T2, T3>;
    return base.Equals(rightT) && F.DSEquals(this.Item3, rightT.Item3);
}

I always get complaints when I show Equals methods that throw if null is passed in, but I stand by my logic, that the presence of null for these categories of ‘structurally equal’ classes is symptom of an error and I want to be notified. Returning false doesn’t do that.

internal static bool DSEquals<T>(T left, T right) {
    if (left == null && right == null)
        return true;
    if (left == null && right != null)
        return false;
    var len = left as IEnumerable;
    var ren = right as IEnumerable;
    if (len == null && ren == null)
        return left.Equals(right);
    if (len == null && ren != null)
        return false;
    if (len != null && ren == null)
        return false;
    return SequenceEqual(len, ren);
}

DSEquals check the content of the Tuple and forward to SequenceEqual in case one slot of the Tuple contains an IEnumerable.

internal static bool SequenceEqual(IEnumerable en1, IEnumerable en2) {
    var enumerator = en2.GetEnumerator();
    foreach (var o in en1) {
        if (!enumerator.MoveNext())
            return false;
        if (!DSEquals(o, enumerator.Current))
            return false;
    }
}

SequenceEqual checks that the number of items in the enumerator is the same and recursively calls DSEqual to check structural equality for items at the same index in the two enumerators.

GetHashCode is trivial (and maybe trivially wrong, one of these days I’ll learn everything about GetHashCode() ).

public override int GetHashCode() {
    return base.GetHashCode() | Item3.GetHashCode();
}

The equality operators are equally simple.

public static bool operator ==(Tuple<T1, T2, T3> t1, Tuple<T1, T2, T3> t2) {
    Utils.CheckNull(t1);
    Utils.CheckNull(t2);
    return t1.Equals(t2);
}
public static bool operator !=(Tuple<T1, T2, T3> t1, Tuple<T1, T2, T3> t2) {
    return !(t1 == t2);
}

And ToString() prints my favorite Tuple format.

public override string ToString() {
    return string.Format("{0},{1}", base.ToString(), Item3.ToString());
}

I’m sure you can find plenty of issues in this code. As always, it is not ‘production ready’, it is more ‘Luca having fun doing it’. In any case, there are some testcases in the solution to check the extent of my testing.

In the next post we’ll look at Records.

Functional.zip

A C# library to write functional code - Part I - Background

Tue, 01 Apr 2008 13:36:00 GMT

Other posts in the series:

Part I - Background
- Part II - Tuples
  - Part III - Records
  - Part IV - Type Unions
  - Part V - The Match operator In December (slow time in msft) I decided to understand what functional programming is all about. When I say ‘understanding’ I don’t mean just paying lip service to the main concepts by knowingly mentioning them in casual conversations (i.e. look at this memoization, man! or this lambda function is so hot!. I can already do that. I intellectually know what the thing is.
  I wanted to *really* understand it. For me that means writing plenty of code. I had a medium size application in my mind that I’ve been wanting to write for quite some time (stock price, dividends, splits downloading and various return calculations), so I went ahead and wrote it. I also wanted to use C#. It would have been easier in F#, but I work on the C# team and love using our own product.
  
  My early attempts were unpleasing. I would fall back to my OO background and my functional code slowly reverted to OO code. My way of thinking about it, even if starting with the best intentions, would go back to: what are the objects, what are their responsibilities and such.
  
  I needed to force myself somehow; kind of overcompensate on the other side. I hit on the idea of pragmatically defining functional programming and try to limit myself to the subset of language constructs inside my definition. As a way to define it, I used Chapter 3 of Expert F#. I know, I know, I could have read 1,000s of academic papers and come up with a meta-analysis of all of them that formally defines what ‘functional programming’ really is. But life is too short. I trusted Don.
  
  The problem is, several of the language constructs in my small definition of functional programming don’t exist in C#. So I went ahead and created them. I built a little library to represent them and forced myself to write code using just this library. It worked.
  
  In this series of posts I will describe what’s inside this library. I want to emphasize that I built it for educational purpose only, not for performance or production code. Caveat emptor.
  
  My plan is to cover the following:
  1. Tuples
    - Records
      - Type Unions
        
        Match
  Let’s see if I can find the time to actually write these posts 🙂

LINQ to SQL Overview - video

Mon, 03 Mar 2008 16:51:00 GMT

This is my presentation from TechEd Barcelona: The link has been fixed. It should work now. Let me know if it doesn’t. http://www.microsoft.com/emea/msdn/spotlight/sessionh.aspx?videoid=716. The European locale brings out my Italian accent

On this page there are a bunch of interesting presentations (i.e. Anders on LINQ): http://www.microsoft.com/emea/msdn/spotlight/default.aspx

Creating an immutable value object in C# - Part V - Using a library

Fri, 11 Jan 2008 13:36:00 GMT

Other posts:

In the last post we presented a variation of implementing a value object using a class. Everything works (obviously), but the amount of code to write is unpleasing. In this post we examine a library based solution. I just describe how to use the Record class, not how it is implemented. You can read the attached implementation code (it is in functional.cs). There is much more in there than Record<>. I’ll talk about the rest in a (hopefully) upcoming series.

To use the record class you need to inherit from it, as in:

public class DateSpan: Record<DateTime, DateTime, bool> {...}

The generic argument types represent the types that comprise the (immutable) state of the object. You then need a friendly way for folks to access this state:

public DateTime Start { get { return state.Item1; } }
    public DateTime End { get { return state.Item2; } }
    public bool HasValue { get { return state.Item3; } }

This is all you have to do. You don’t need to implement Equals, ==, != and GetHashCode. Structural equivalence is given to you by the Record class. Such a property is recursive, in the sense that you can embed value objects inside other value objects and the implementation would walk your object graph as necessary.

For example, given the following class hierarchy:

public class Order: Record<string, int> {
        public Order(string item, int qty): base(item,qty) {}
        public string Item { get { return state.Item1;}}
        public int Quantity { get { return state.Item2; } }
    }
    public class Customer: Record<string, IEnumerable<Order>> {
        public Customer(string name, IEnumerable<Order> orders) : base(name, orders) { }
        public string Name { get { return state.Item1; } }
        public IEnumerable<Order> Orders { get { return state.Item2; } }
    }

The following test case succeed:

[TestMethod]
        public void Record2Test() {
            var c1 = new Customer("Luca", new Order[] { new Order("car",1), new Order("stereo", 3)});
            var c11 = new Customer("Luca", new Order[] { new Order("car", 1), new Order("stereo", 3) });
            var c2 = new Customer("Bob", new Order[] { new Order("car", 1), new Order("stereo", 3) });
            var c3 = new Customer("Bob", new Order[] { new Order("car", 1), new Order("stereo", 2) });
            Assert.AreEqual(c1, c11);
            Assert.AreNotEqual(c1, c2);
            Assert.AreNotEqual(c1, c3);
            Assert.AreNotEqual(c2, c3);
        }

Please don’t take my library as production ready code. The amount of test I put into it is limited. You can probably find obvious bugs with it.

Let’s look at other drawbacks. The biggest one is that I’m stealing your base class. If you want your value object to inherit from something else, you cannot. You cannot even have value objects inherit from each other. In that case you are back to implementing your own Equals, == and so on. The only tools at your disposal are interfaces and composition.

Another drawback is that writing classes in this way is slightly unnatural. You have to think about the ‘type’ of your state in the declaration of the class itself instead of more naturally writing it closer to where you assign names to it (property/field declaration).

Having considered these drawbacks, I’m using this library in all my code wherever I need value objects (which is almost everywhere these days). Writing all the Equals explicitly is too error prone for my taste. I will also be creating IDE snippets for myself that make writing these classes easier.

I don’t think I have anything else to say on this topic, so this will be my last post on it. If something else comes up, I’ll let you know.

TimeLineUsingRecord.zip

Book review: The Halo Effect

Mon, 07 Jan 2008 14:04:22 GMT

When I read Built to Last, In search of excellent and Good to great I immediately thought: What a bunch of BS!!. But again, I think that about most business books.

The content of these books seemed particularly ludicrous to me. I found both logical inconsistencies and methodological flaws in their process. For example, picking successful companies after the fact is analogous to picking the winners of a lottery and claim that they have exceptional skills. Betting the farm on a single concept (i.e. Big Hairy Goal) is clearly the right thing when you are right, but most often than not you are wrong. What about all the companies that did that and failed? Also, Keeping to the Core worked really well for Coca Cola, but really badly for Kodak.

These are just examples. These books are packed full of such reasonably sounding absurdities. I thought that everybody would immediately spot this. I thought that the mediocre subsequent performance of the companies originally selected as excellent would obviously demonstrate the unsoundness of such conclusion. I was very wrong. These tomes are hugely successful. People seemed to disagree with me. They really like this stuff. Given that my work and my life were not overly impacted one way or the other, I left the matter to rest.

And then came The Halo Effect. It was a pleasure for me to read, as it confirmed my own ideas (confirmation bias anyone?) and added many more.

The title is slightly misleading. A better title would have been: Why the idea of finding common traits among successful companies is baloney. I know, not as catchy. The catchy halo effect, mentioned in the title, is just the biggest delusion that people have about the topic.

I found an experiment recounted in the book to be very illuminating. They divided a set of people in multiple groups and gave each group a problem to solve. They then told half of the groups that they did a very good job and the other half that they did a very poor job. The thing is, they did so at random. They then asked people to rate the dynamics of their group on several factors. People that were told that they did a good job reported to have had wonderful dynamics: open discussions, inspired leaders, high caliber people and such. The opposite was true for the other groups.

It didn’t matter what the actual dynamics had really been: some groups were very confrontational, others were very amicable; some were very vocal, others were very quiet. In the end, people who were told that they did good recalled the experience in positive terms. People who were told that they did poorly, recalled the experience in negative terms. Remember, good and bad outcomes were assigned randomly. Also, the results have been confirmed in several similar studies.

It is not a good strategy that drives performance. It is perceived performance that makes people think that they had a good strategy. This is: The halo effect. This simple concept has dramatic repercussions.

First of all, it invalidates the main process used in these studies: if you use newspapers articles and managers recollection to draw conclusions, you are just piling halos over halos. People would certainly give all sort of positive attributes to companies that succeeded. They do it exactly because they succeeded. And when these companies fail, they suddenly give all sort of negative attributes to them. The book is full of examples of companies that were rated as excellent when their stock price went up and terrible when their stock price went down. Nothing changed in these companies, they were still doing the same exact thing.

The book contains many other insights of why these ‘studies’ are deeply flawed. It calls them delusions. You are probably familiar with some of them if you have a scientific background: for example, Delusion of Correlation and Causality, Delusion of Single Explanation (factors are not independent), Delusion of Connecting the Winning Dots (ad hoc after the fact selection). Other delusions are specific to the analyzed domain: Delusion of Lasting Success, Delusion of Absolute Performance, Delusion of Organizational Physics.

In the end what should managers do? Instead of following these books’s precepts, they should focus on two things: strategy and execution. Both of these things are probabilistic endeavors where the absolutely wrong measuring stick is the result that you obtain. I always found this to be a very common error people fall into at whatever level in an organization. The error is judging the soundness of a decision by its result. A decision should be judged by the soundness of the process to get to the decision and the information available at the time the decision was taken. This is the only thing that is consistent with the probabilistic nature of the world we live in. The book profiles three leaders who embody this probabilistic view.

In summary, please go and read this book. There are many more things in it than I described in this (albeit long and boring) post.

Creating an immutable value object in C# - Part IV - A class with a special value

Fri, 28 Dec 2007 18:45:32 GMT

Other posts:

In the last post we looked at structs as ways to implement immutable value objects and we discovered that they have several issues.

A good thing about the struct implementation was the introduction of an explicit ‘special value’ instead of ‘null’. I personally like doing that because it forces me to think about what are the special values in my domain instead of blindly rely on null and its semantics. Plus, it also works where null breaks down, when there are multiple special values.

Having explicit special values is obviously not related to structs in any way. You can use the same pattern with classes as well, but you have to manage ‘null’ values that can enter your API. Here is how I did it for the DateSpan class.

First I defined an utility function to manage null values:

public static void CheckNull<T>(T t) {
    if (t == null)
        throw new ArgumentNullException();
}

Then I had to check my null precondition for each parameter of my API. I.E.

public DateSpan Intersect(DateSpan other) {
    Utils.CheckNull(other);
    ...
}

Most importantly I now have to check for null in ‘Equals’ and ‘==’:

public override bool Equals(object obj) {
    // Interesting choice, null is not valid in this domain
    Utils.CheckNull(obj);
    if (this.GetType() != obj.GetType()) return false;
    DateSpan other = obj as DateSpan;
    return other.End == End && other.Start == Start;
}

public static Boolean operator ==(DateSpan v1, DateSpan v2) {
    Utils.CheckNull(v1);
    Utils.CheckNull(v2);
    return (v1.Equals(v2));
}

So now we have an immutable value object, represented as a class, with checks for nulls and a special value (not shown above because it is essentially the same as for structs). So, does this work?

It does, but it is cumbersome to write. And if it is too cumbersome, I already know that I’m not going to use it. I might start doing it, but I would soon give up. Are there ways to ease the pain?

One way would be to use snippets to help in writing the code. Snippets in this case have a couple of problems:

It is not easy to ‘snippify’ the logic inside ‘Equals’, ‘GetHashcode’ and such
It makes easy to write the code, but still it is hard to read it and maintain it

In the next post we’ll look at a better solution.

Creating an immutable value object in C# - Part III - Using a struct

Mon, 24 Dec 2007 17:39:00 GMT

Other posts:

In Part II I talked about the asymmetry created by using ‘null’ as the special value for our little DateSpan domain. We also noticed the boredom of having to implement Equals, GetHashCode, ‘==’ and ‘!=’ for our value objects. Let’s see if structs solve our problem.

Well, to the untrained eye they do. Structs cannot be null and they implement Equals and GetHashCode by checking the state of the object, not its pointer in memory.

So, have we found the perfect tool to implement our value object?

Unfortunately, no. Here is why a struct is a less than optimal way to implement a value object:

Convenience issues - it is not as convenient as it looks
You still have to implement ‘==’ and ‘!=’
You still want to implement Equals() and GetHashCode(), if you want to avoid boxing/unboxing.
Performance issues - it is not as fast as it looks
Structs are allocated on the stack. Every time you pass them as arguments, the state is copied. If your struct has more than a few fields, performance might suffer
Usability issues - it is not as useful as it looks.
Structs always have a public default constructor that ‘zeros’ all the fields
Structs cannot be abstract
Structs cannot extend another structs

Don’t get me wrong, structs are extremely useful as a way to represent small bundles of data. But if you use value objects extensively, their limitations start to show.

A case could be made that you should use struct to implement value objects if the issues exposed above don’t apply to your case. When they do apply, you should use classes. I’m a forgetful and lazy programmer, I don’t want to remember all these cases. I just want a pattern that I can use whenever I need a value object. It seems to me that structs don’t fit the bill.

For the sake of completeness, here is the code for DateSpan using a struct. Note that I explicitly introduced a ‘special value’ instead of using null (which is not available for structs).

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

public struct DateSpan {
    public static DateSpan NoValueDateSpan { get { return noValueDateSpan; } }
    public DateSpan(DateTime pstart, DateTime pend) {
        if (pend < pstart)
            throw new ArgumentException(pstart.ToString() + " doesn't come before " + pend.ToString());
        start = pstart;
        end = pend;
        hasValue = true;
    }
    public DateSpan Union(DateSpan other) {
        if (!HasValue)
            return other;
        if (!other.HasValue)
            return this;
        if (IsOutside(other))
            return DateSpan.NoValueDateSpan;
        DateTime newStart = other.Start < Start ? other.Start : Start;
        DateTime newEnd = other.End > End ? other.End : End;
        return new DateSpan(newStart, newEnd);
    }
    public DateSpan Intersect(DateSpan other) {
        if (!HasValue)
            return DateSpan.NoValueDateSpan;
        if (!other.HasValue)
            return DateSpan.NoValueDateSpan;
        if (IsOutside(other))
            return DateSpan.NoValueDateSpan;
        DateTime newStart = other.Start > Start ? other.Start : Start;
        DateTime newEnd = other.End < End ? other.End : End;
        return new DateSpan(newStart, newEnd);
    }
    public DateTime Start { get { return start; } }
    public DateTime End { get { return end; } }
    public bool HasValue { get { return hasValue; } }
    // Making field explicitely readonly (but cannot use autoproperties)
    // BTW: If you want to use autoproperties, given that it is a struct,
    // you need to add :this() to the constructor
    private readonly DateTime start;
    private readonly DateTime end;
    private readonly bool hasValue;
    private bool IsOutside(DateSpan other) {
        return other.start > end || other.end < start;
    }
    // Changing the internal machinery so that hasValue default is false
    // This way the automatically generated empty constructor returns the right thing
    private static DateSpan noValueDateSpan = new DateSpan();
    #region Boilerplate Equals, ToString Implementation
    public override string ToString() {
        return string.Format("Start:{0} End:{1}", start, end);
    }
    public static Boolean operator ==(DateSpan v1, DateSpan v2) {
        return (v1.Equals(v2));
    }
    public static Boolean operator !=(DateSpan v1, DateSpan v2) {
        return !(v1 == v2);
    }
    //public override bool Equals(object obj) {
    //    if (this.GetType() != obj.GetType()) return false;
    //    DateSpan other = (DateSpan) obj;
    //    return other.end == end && other.start == start;
    //}
    //public override int GetHashCode() {
    //    return start.GetHashCode() | end.GetHashCode();
    //}
    #endregion
}

TimeLineAsStruct.zip

Bisection-based XIRR implementation in C#

Mon, 17 Dec 2007 18:17:00 GMT

Here is a quick implementation of XIRR (using Excel nomenclature) written in C#.

Disclaimer: this is a super simple Bisection-based implementation. People tend to prefer the Newton method, but this is simpler and works for the app I’m writing. I decided to post it because I couldn’t find one on the net when I looked for it. I attached testcases to show the extent of my testing.

It is called CalculateXIRR and it is invoked by passing a list of cash flows, a tolerance and a max number of iterations.

using System;

using System.Linq;

using Money = System.Decimal;

using Rate = System.Double;

using System.Collections.Generic;
public

 struct Pair<T, Z> {
public Pair(T first, Z second) { First = first; Second = second; }

public readonly T First;

public readonly Z Second;

}


public class CashFlow {

public CashFlow(Money amount, DateTime date) { Amount = amount; Date = date; }

public readonly Money Amount;
public readonly DateTime Date;
}

public struct AlgorithmResult<TKindOfResult, TValue> {

public AlgorithmResult(TKindOfResult kind, TValue value) {

Kind = kind;
Value = value;
}

public readonly TKindOfResult Kind;
public readonly TValue Value;
}

public enum ApproximateResultKind {
ApproximateSolution,
ExactSolution,
NoSolutionWithinTolerance
}

public static class Algorithms {

internal static Money CalculateXNPV(IEnumerable<CashFlow> cfs, Rate r) {

if (r <= -1)
r= -0.99999999; // Very funky ... Better check what an IRR <= -100% means

return (from cf in cfs
let startDate = cfs.OrderBy(cf1 => cf1.Date).First().Date
select cf.Amount / (decimal) Math.Pow(1 + r, (cf.Date - startDate).Days / 365.0)).Sum();
}

internal static Pair<Rate, Rate> FindBrackets(Func<IEnumerable<CashFlow>, Rate, Money> func, IEnumerable<CashFlow> cfs) {

// Abracadabra magic numbers ...
const int maxIter = 100;
const Rate bracketStep = 0.5;
const Rate guess = 0.1;

Rate leftBracket = guess - bracketStep;
Rate rightBracket = guess + bracketStep;
var iter = 0;

while (func(cfs, leftBracket) * func(cfs, rightBracket) > 0 && iter++ < maxIter) {

leftBracket -= bracketStep;
rightBracket += bracketStep;
}

if (iter >= maxIter)
return new Pair<double, double>(0, 0);

return new Pair<Rate, Rate>(leftBracket, rightBracket);
}

// From "Applied Numerical Analyis" by Gerald
internal static AlgorithmResult<ApproximateResultKind, Rate> Bisection(Func<Rate, Money> func, Pair<Rate, Rate> brackets, Rate tol, int maxIters) {

int iter = 1;

Money f3 = 0;
Rate x3 = 0;
Rate x1 = brackets.First;
Rate x2 = brackets.Second;

do {
var f1 = func(x1);
var f2 = func(x2);

if (f1 == 0 && f2 == 0)
return new AlgorithmResult<ApproximateResultKind, Rate>(ApproximateResultKind.NoSolutionWithinTolerance, x1);

if (f1 * f2 > 0)
throw new ArgumentException("x1 x2 values don't bracket a root");

x3 = (x1 + x2) / 2;
f3 = func(x3);

if (f3 * f1 < 0)
x2 = x3;
else
x1 = x3;

iter++;

} while (Math.Abs(x1 - x2)/2 > tol && f3 != 0 && iter < maxIters);

if (f3 == 0)
return new AlgorithmResult<ApproximateResultKind, Rate>(ApproximateResultKind.ExactSolution, x3);

if (Math.Abs(x1 - x2) / 2 < tol)
return new AlgorithmResult<ApproximateResultKind, Rate>(ApproximateResultKind.ApproximateSolution, x3);

if (iter > maxIters)
return new AlgorithmResult<ApproximateResultKind, Rate>(ApproximateResultKind.NoSolutionWithinTolerance, x3);

throw new Exception("It should never get here");
}

public static AlgorithmResult<ApproximateResultKind, Rate> CalculateXIRR(IEnumerable<CashFlow> cfs, Rate tolerance, int maxIters) {

var brackets = FindBrackets(CalculateXNPV, cfs);

if (brackets.First == brackets.Second)
return new AlgorithmResult<ApproximateResultKind, double>(ApproximateResultKind.NoSolutionWithinTolerance, brackets.First);

return Bisection(r => CalculateXNPV(cfs,r), brackets, tolerance, maxIters);
}
}

// TESTS
using Microsoft.VisualStudio.TestTools.UnitTesting;

using System.Collections.Generic;

using System;

using Rate = System.Double;
namespace TimeLineTest
{
[TestClass()]
public class AlgorithmsTest {

IEnumerable<CashFlow> cfs = new CashFlow[] {
new CashFlow(-10000, new DateTime(2008,1,1)),
new CashFlow(2750, new DateTime(2008,3,1)),
new CashFlow(4250, new DateTime(2008,10,30)),
new CashFlow(3250, new DateTime(2009,2,15)),
new CashFlow(2750, new DateTime(2009,4,1))
};

IEnumerable<CashFlow> bigcfs = new CashFlow[] {
new CashFlow(-10, new DateTime(2000,1,1)),
new CashFlow(10, new DateTime(2002,1,2)),
new CashFlow(20, new DateTime(2003,1,3))
};

IEnumerable<CashFlow> negcfs = new CashFlow[] {
new CashFlow(-10, new DateTime(2000,1,1)),
new CashFlow(-1, new DateTime(2002,1,2)),
new CashFlow(1, new DateTime(2003,1,3))
};

IEnumerable<CashFlow> samedaysamecfs = new CashFlow[] {
new CashFlow(-10, new DateTime(2000,1,1)),
new CashFlow(10, new DateTime(2000,1,1)),
};

IEnumerable<CashFlow> samedaydifferentcfs = new CashFlow[] {
new CashFlow(-10, new DateTime(2000,1,1)),
new CashFlow(100, new DateTime(2000,1,1)),
};

IEnumerable<CashFlow> bigratecfs = new CashFlow[] {
new CashFlow(-10, new DateTime(2000,1,1)),
new CashFlow(20, new DateTime(2000,5,30)),
};

IEnumerable<CashFlow> zeroRate = new CashFlow[] {
new CashFlow(-10, new DateTime(2000,1,1)),
new CashFlow(10, new DateTime(2003,1,1)),
};

IEnumerable<CashFlow> doubleNegative = new CashFlow[] {
new CashFlow(-10000, new DateTime(2008,1,1)),
new CashFlow(2750, new DateTime(2008,3,1)),
new CashFlow(-4250, new DateTime(2008,10,30)),
new CashFlow(3250, new DateTime(2009,2,15)),
new CashFlow(2750, new DateTime(2009,4,1))
};

IEnumerable<CashFlow> badDoubleNegative = new CashFlow[] {
new CashFlow(-10000, new DateTime(2008,1,1)),
new CashFlow(2750, new DateTime(2008,3,1)),
new CashFlow(-4250, new DateTime(2008,10,30)),
new CashFlow(3250, new DateTime(2009,2,15)),
new CashFlow(-2750, new DateTime(2009,4,1))
};

double r = 0.09;
double tolerance = 0.0001;
int maxIters = 100;

private TestContext testContextInstance;

public TestContext TestContext {
get {
return testContextInstance;
}
set {
testContextInstance = value;
}
}

[TestMethod()]
public void CalculateXNPV() {

Assert.AreEqual(2086.6476020315416570634272814M, Algorithms.CalculateXNPV(cfs, r));
Assert.AreEqual(-10.148147600710372651326920258M, Algorithms.CalculateXNPV(negcfs, 0.5));
Assert.AreEqual(4.9923725815954514810351876895M, Algorithms.CalculateXNPV(bigcfs, 0.3));
}

[TestMethod]
public void FindBrackets() {

var brackets = Algorithms.FindBrackets(Algorithms.CalculateXNPV, cfs);
Assert.IsTrue(brackets.First < 0.3733 && brackets.Second > 0.3733);

brackets = Algorithms.FindBrackets(Algorithms.CalculateXNPV, bigcfs);
Assert.IsTrue(brackets.First < 0.5196 && brackets.Second > 0.5196);

brackets = Algorithms.FindBrackets(Algorithms.CalculateXNPV, negcfs);
Assert.IsTrue(brackets.First < -0.6059 && brackets.Second > -0.6059);
}

[TestMethod]
public void XIRRTest() {

var irr = Algorithms.CalculateXIRR(cfs, tolerance, maxIters);
Assert.AreEqual(0.3733, irr.Value, 0.001);
Assert.AreEqual(ApproximateResultKind.ApproximateSolution, irr.Kind);

irr = Algorithms.CalculateXIRR(bigcfs, tolerance, maxIters);
Assert.AreEqual(0.5196, irr.Value, 0.001);
Assert.AreEqual(ApproximateResultKind.ApproximateSolution, irr.Kind);

irr = Algorithms.CalculateXIRR(negcfs, tolerance, maxIters);
Assert.AreEqual(-0.6059, irr.Value, 0.001);
Assert.AreEqual(ApproximateResultKind.ApproximateSolution, irr.Kind);

irr = Algorithms.CalculateXIRR(samedaysamecfs, tolerance, maxIters);
Assert.AreEqual(ApproximateResultKind.NoSolutionWithinTolerance, irr.Kind);

irr = Algorithms.CalculateXIRR(samedaydifferentcfs, tolerance, maxIters);
Assert.AreEqual(ApproximateResultKind.NoSolutionWithinTolerance, irr.Kind);

irr = Algorithms.CalculateXIRR(bigratecfs, tolerance, maxIters);
Assert.AreEqual(4.40140, irr.Value, 0.001);
Assert.AreEqual(ApproximateResultKind.ApproximateSolution, irr.Kind);

irr = Algorithms.CalculateXIRR(zeroRate, tolerance, maxIters);
Assert.AreEqual(0, irr.Value, 0.001);
Assert.AreEqual(ApproximateResultKind.ApproximateSolution, irr.Kind);

irr = Algorithms.CalculateXIRR(doubleNegative, tolerance, maxIters);
Assert.AreEqual(-0.537055, irr.Value, 0.001);
Assert.AreEqual(ApproximateResultKind.ApproximateSolution, irr.Kind);

irr = Algorithms.CalculateXIRR(badDoubleNegative, tolerance, maxIters);
Assert.AreEqual(ApproximateResultKind.NoSolutionWithinTolerance, irr.Kind);
}
}
}

Creating an immutable value object in C# - Part II - Making the class better

Thu, 06 Dec 2007 12:34:00 GMT

Other posts:

In the previous post I showed how to trivially implement a value object. The code works but it has several issues. Some are very simple, others are more interesting.

Let’s take a look at them:

State not explicitly read-only
Asymmetry in the usage of Union and Intersection
Small perf issue in the Union method

The first problem is that my use of automatic properties doesn’t assure that the status of the object is immutable; I can still modify it from inside the class. The simple solution is to make the fields readonly and write the getters as in:

private readonly DateTime start;
    private readonly DateTime end;
    public DateTime Start { get { return start; } }
    public DateTime End { get { return end; } }

The second issue is more subtle. Consider this code

DateSpan d1 = new DateSpan(new DateTime(1, 1, 2000), new DateTime(1, 1, 2002));
        DateSpan d2 = null;
        DateSpan r1 = d1.Intersect(d2);
        DateSpan r2 = d2.Intersect(d1);

I would like things in my little ‘algebra’ to be symmetric. In this case I’d like: r1 == r2 == null. But this code throws in the last line as I’m trying to invoke a method on a null object.

The traditional solution to this problem is to make Union and Intersect to be static methods, but then you loose the magic of calling them as instance methods (i.e. it becomes hard to chain them together as in d1.Intersect(d2).Union(d3).Intersect(d4)).

Extension methods come to the rescue here as they allow you to create static methods, but to call them as if the were instance methods. The new code for Intersect looks like the following:

public static DateSpan Intersect(this DateSpan one, DateSpan other) {
        if (one == null)
            return null;
        if (other == null)
            return null;
        if (one.IsOutside(other))
            return null;
        DateTime start = other.Start > one.Start ? other.Start : one.Start;
        DateTime end = other.End < one.End ? other.End : one.End;
        return new DateSpan(start, end);
    }

This workaround would not work if the extension method needs to access private state of the class. In that case you would need to create a static method on the DataSpan class and invoke it from the extension method. Slightly more convoluted, but still doable.

At a more philosophical level, the asymmetry issue happens here because I’m using something outside my domain of interest (the null value) to represent a special value inside my domain of interest. More on this as we talk about structs in upcoming posts.

The last point is a very small one. In the Union method I am creating a new object unnecessarily in the following line:

if (other == null)
            return new DateSpan(Start, End);

I can obviously avoid it by just returning this.

This post hints to several possible issues. Is it a good idea to use null to represent special values in my domain? What if I have more than one of them (i.e. positive/negative infinite)? Would using structs solve these problems?

We’ll take a look at these options in upcoming posts. Attached is the modified code.

Creating an immutable value object in C# - Part I - Using a class

Mon, 03 Dec 2007 13:22:00 GMT

Other posts:

Value objects are objects for which the identity is based on their state instead of their pointer in memory. For example, a numeric Complex class is, most of the time, a value object because you can treat two instances as the same if their state (real and img fields in this case) is the same. An immutable value object is a value object that cannot be changed. You cannot modify its state, you have to create new ones.

I’m using these guys more and more in my code for a number of reasons, both practical and philosophical. The practical ones revolve around the greater robustness of programming without side effects and the greater simplicity of parallelizing your code. The philosophical ones are more interesting (and subjective). When in my design process I spend the time to aggressively looking for these kinds of objects, the resulting design ends up cleaner. I especially like when I can define some sort of close algebra for these guys (i.e. a set of functions that operate over them and produces new ones, not unlike ‘+’ and ‘-’ for numbers).

This series describes how to create immutable value objects in C# and the design decisions involved. This is a summary of an email thread I had with Mads and Luke.

The concept I will use for this series is a DateSpan. As I define it, a DateSpan has a Start and an End date. You can ask for the DateSpan that represents the Union and Intersection of two DateSpans. The tests in the attached code better define the behavior of these operations.

Given that I never use structs in my code (I’m a minimalist language user), I’ll start by using a class to represent it. We’ll make this class better in part II and use a struct in part III. A first stab at it is as follows:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
public class DateSpan {
    public DateSpan(DateTime start, DateTime end) {
        if (end < start)
            throw new ArgumentException(start.ToString() + " doesn't come before " + end.ToString());
        Start = start;
        End = end;
    }
    public DateSpan Union(DateSpan other) {
        if (other == null)
            return new DateSpan(Start, End);
        if (IsOutside(other))
            return null;
        DateTime start = other.Start < Start ? other.Start : Start;
        DateTime end = other.End > End ? other.End : End;
        return new DateSpan(start, end);
    }
    public DateSpan Intersect(DateSpan other) {
        if (other == null)
            return null;
        if (IsOutside(other))
            return null;
        DateTime start = other.Start > Start ? other.Start : Start;
        DateTime end = other.End < End ? other.End : End;
        return new DateSpan(start, end);
    }
    private bool IsOutside(DateSpan other) {
        return other.Start > End || other.End < Start;
    }
    public DateTime Start { get; private set; }
    public DateTime End { get; private set; }
    #region Boilerplate Equals, ToString Implementation
    public override string ToString() {
        return string.Format("Start:{0} End:{1}", Start, End);
    }
    public override bool Equals(object obj) {
        if (obj == null) return false;
        if (this.GetType() != obj.GetType()) return false;
        DateSpan other = obj as DateSpan;
        return other.End == End && other.Start == Start;
    }
    public override int GetHashCode() {
        return Start.GetHashCode() | End.GetHashCode();
    }
    public static Boolean operator ==(DateSpan v1, DateSpan v2) {
        if ((object)v1 == null)
            if ((object)v2 == null)
                return true;
            else
                return false;
        return (v1.Equals(v2));
    }
    public static Boolean operator !=(DateSpan v1, DateSpan v2) {
        return !(v1 == v2);
    }
    #endregion
}

public static class TimeLineExtensions {
    public static bool IsSuperSet(this DateSpan span, DateSpan other) {
        if (span.Intersect(other) == other)
            return true;
        return false;
    }
}

Some things to notice in this code:

Defining a value object with a C# class involves overriding Equals, Hashcode, == and !=. It is tricky, but usually boilerplate stuff, well described in Wagner (2004). I don’t have Bill’s book in my office, so I kind of made it up on the fly. It could be very wrong (the ‘==’ one looks very suspicious). Don’t copy it, read Bill’s book instead
Defining an immutable object with a C# class involves discipline in not changing the private state (we’ll see in Part II that we can do better than ‘discipline’)
Notice the extension method IsSuperSet. This is something I picked up from an old Coplien book, probably this one. The concept is to keep methods that don’t use internal state of an object external to the object itself for the sake of future maintainability. The syntax for doing that was awkward before, but extension methods make it easier

This works (it passes all the tests), but it has a number of problems. We’ll talk about these in Part II.

TimeLineAsClass - 1.zip

Brief review of Total Workday Control

Tue, 20 Nov 2007 11:56:46 GMT

We get plenty of emails in Microsoft. Without a system to manage them you are dead. Or better, your mind plays strange tricks on you making you think that you still have work to do. You need to convince your mind that you are done for the day and to stop bugging you, please.

Almost everyone has a system. Some are better than others. My current system is Getting things done and it does achieve its goal of freeing my mind (mostly) without imposing too much overhead. But I’m always looking for something better. That’s why I checked out Total Workday Control.

The title conjures in my mind all sort of scary images of control freaks going through their days trying to achieve optimization of every single action. In reality the book is just a light update of David Allen methodology. The most interesting thing I picked out is the differentiation between a Master Task List and a Daily List. The former contains projects or delayed tasks while the latter contains tasks (typically taken from the Master List) that you want to perform today. It also has a system of prioritizing such tasks and give good rules of thumbs on how many you should have on each list and so on.

My rule of thumb is that if I learn one useful thing by reading a book, then it is a fantastic book. This book qualifies. If you are currently using Getting things done, this book constitutes a nice upgrade. If you aren’t, I suggest you read David Allen first, as it gives you the reasons for the system, before you move to the more technical Total Workday Control.

Instead of a simple switch statement

Fri, 31 Aug 2007 16:54:15 GMT

This is Luke‘s kind of code. I might be catching the virus

```csharp abstract class QIFParserBase { public enum LoadOptions { All, Prices, Securities, Transactions } static readonly Dictionary> parseFuncs = new Dictionary> { {LoadOptions.All, (q,c) => q.ParseAll(c)}, {LoadOptions.Prices, (q,c) => q.ParsePricesBlocks(c)}, {LoadOptions.Securities, (q,c) => q.ParseSecurityBlocks(c)}, {LoadOptions.Transactions, (q,c) => q.ParseTransactionBlocks(c)} }; public QIFParserBase(string fileName, LoadOptions opt) { string content = File.ReadAllText(fileName); string[] blocks = content.Split(new string[] { “!Type:”, “!Option:” }, StringSplitOptions.RemoveEmptyEntries); parseFuncsopt; }

Parsing QIF Quicken files in C#

Fri, 31 Aug 2007 16:43:00 GMT

I’m slightly prouder of the structure of this code than the one in the previous blog post. You can simply inherit from QIFParserBase and override a couple of abstract methods to customize the behavior. Still, I just tested it on a couple of test QIF files. It is not production quality at all.

Notice that I don’t even have Quicken. I’m producing these test file with FundManager, which I use for my investments. If your software generates QIF files differently, than you have to modify the code. It shouldn’t be too hard.

It works with VS 2008 beta 2.

ParseQIF.zip

Retrieve prices, dividends and splits for a stock in C#

Thu, 30 Aug 2007 16:25:00 GMT

I wrote this code very quickly and I’m kind of ashamed of it, but it gets the job done (I think). You need the HTML Agility Pack for the stock splits retrieving code. You can download it from here or you can simply comment out the code. I wrote it against Visual Studio 2008 beta 2, but it should be trivial to port it to previous versions.

You run it from a command window like this: priceretriever msft 1/1/1990 2/3/2003. The last two parameters are optional and default to 1/1/1980 and today.

Enjoy.

PriceRetriever.zip

Lucabol and Kiraly

Mon, 13 Aug 2007 20:42:00 GMT

My wife took this in Manhattan Beach this week end. Karch is the one on the right.

BeachVolley 123.JPG

Video of Matt and I talking about the LINQ to SQL query/update pipeline

Mon, 13 Aug 2007 12:56:00 GMT

And a bit of history of O/R in Microsoft at the end: http://blogs.msdn.com/charlie/archive/2007/08/06/linq-to-sql-pipeline-video-with-luca-bolognese-and-matt-warren.aspx

Interesting 10 years retrospective on The cathedral and the Bazaar

Thu, 05 Jul 2007 12:12:00 GMT

Interesting read on what open source has been good for, on the difference between Linux and Wikipedia and on how to best harness the power of crowds.

Also a highly positively reviewed new book that formalizes in almost mathematical fashion what diversity is good for (I haven’t read it yet). The classical work on the matter is “The Wisdom of crowds”.

Excerpt from the initial article: “Raymond was clear on that point when, toward the end of his paper, he examined some of the “necessary preconditions” for the bazaar model of production. “It’s fairly clear,” he wrote, “that one cannot code from the ground up in bazaar style. One can test, debug and improve in bazaar style, but it would be very hard to originate a project in bazaar mode.” In a recent e-mail to me, he was even blunter. “The individual wizard,” he wrote, “is where successful bazaar projects generally start.” “

My TechEd webcast is here

Tue, 19 Jun 2007 10:32:00 GMT

In case you want to know more about LINQ: http://www.virtualteched.com/Pages/OnlineSessions/OnlineWebcastsDevTools.aspx

CTRL+F for ‘(LINQ)’

An introduction to the talk is here: http://www.virtualteched.com/videos/luca%20bolognese%20sml.asx

ObjectSpaces early days

Thu, 07 Jun 2007 10:40:00 GMT

Matt has a very good post on the history of object relational framework inside Microsoft. He and I started the whole ObjectSpaces thing together very many years ago (about six and a half). I thought I should add my two cents.

You might be wondering how a project starts inside Microsoft (or you might not). In this case, I was hired in the ADO.NET team to build an object relational framework. That was my assignment. The problem was that I had no dev or tester to work with. I also had just arrived in the US from Italy and my English was terrible (that hasn’t changed much). My only qualifications for the job were that I was incredibly passionate about the topic; I had built a couple of small ones in my spare time and have used many more in various projects. But still I had no idea on how to make things happen in this big company. Hell, I could barely understand what people were talking about at lunch.

Anyhow, my boss told me that there was a guy who was relatively free, but I had to come up with an idea, convince him that the idea was a good one and that he should go ahead and prototype it. That guy was Matt Warren, already one of the best developers in the SQL team and the dev lead of plenty of our .NET stack.

The first time I entered his office and started talking, he immediately told me: “Not now, I’m debugging”. A good start … But after that, we hit it off immediately. We talked about what an OR framework is, what it should be, how we could innovate in the space and so on. After a while, Matt started working on the very first ObjectSpaces prototype. Our modus operandi was peculiar. I would go to Matt’s office and we would discuss a particular feature or scenario. He would tell me: “Ok, we are on the same page, I’ll see you tomorrow”. The next day the feature would be implemented. If he told me: “It is going to take me 5 days to code it” I knew he didn’t like the feature and I had some more convincing to do. Sometime we would agree, sometime we wouldn’t. Sometime I came up with staff to implement, sometimes Matt did. We figured out a lot of stuff in those days.

Also I learned how to work with different categories of programmers. If you work with a bad programmer you have to tell him how to implement something. If you work with a good programmer you have to tell him what you want the feature to look like. If you work with an excellent programmer, you just have to tell him what your final goal is. I quickly realized that the latter was the most productive strategy with Matt. I had just to convince him of the business need for something. Implementing it ended up to be a detail.

That initial prototype grew to become ObjectSpaces (it was called Cheops initially). The team grew to be about 25 people, growing and stretching me personally. We went through two painful rounds of unifications with bigger products. In the end ObjectSpaces was cut. Matt moved to the C# team to work on LINQ. Dinesh and I followed after a short while. The whole ObjectSpaces team scattered in various places around the company. The object relational thing started again in the C# team as part of the LINQ project. This time around we also had a compiler to play with. Plus Anders was on board to sprinkle his design magic over the whole thing.

As Roland would say, the world has moved on. Matt is now a big shot architect and I lead a team of amazingly smart individuals (they are forced to act as if my words make sense, imagine that …). We are shipping a game changing product in LINQ and a wonderful object relational framework in LINQ to SQL. Things turned out for the best (even if five years too late).

Still it is a pleasure to think back at those early days (and nights) of ‘figuring out stuff’ six years ago. A lot of that ‘stuff’ is inside our products today, and that is something to be proud of.

My LINQ Overview session at TechEd will be live on the Internet

Mon, 04 Jun 2007 12:45:00 GMT

at least they told me so. It is at 8.30am Eastern time. I don’t know the exact link, but it will be somewhere here: http://www.microsoft.com/events/teched2007/default.mspx.

Y combinators anyone?

Fri, 11 May 2007 19:16:00 GMT

From Mads, our proud C# language PM: http://blogs.msdn.com/madst/archive/2007/05/11/recursive-lambda-expressions.aspx.

Huntington Beach and Kiraly

Sat, 05 May 2007 22:24:00 GMT

Today I’m watching the AVP tour, especially (as always) Karch. This year is his last and I’m trying to go and see him as much as I can. Karch is an idol for my generation of beach volley players. Well, it is an idol for all generations really

He loses one, he wins one. The day is about to end but he still has to play against Lambo. The sun is going down, it is the last match of the day. The crowd gathers around the court. As always it is not easy to put him down. He plays with Wong, another veteran of the AVP tour. The other team serves him short to force him to move forward and then backward again to prepare his attack. It is a smart strategy, Karch’s legs and not what they used to be 20 years ago

But he goes on, finds new and interesting ways to overcome a block that is way higher than his maximum one hand reach. Sometimes it feels as if the block is his friend and Lambo starts to lower his hands not to give him that tool to play with. But the short serves are wearing him down and the match is going in the wrong direction.

Suddenly, out of nowhere, he produces an amazing strong hit. Something Rosy could have done. The crowd goes ballistic. Everybody laughs and cheers. Even the other team is laughing. Karch raises his hands smiling as to say: I have no idea where that one came from. But maybe he does, because it happens again the very next side out. It looks like something good might still happen. But then he misses the last shot. He tries to hit the line, but the ball is far from the net and his jump is what it is.

We give him a two minutes standing ovation. This is the last time Huntington beach has seen him play. He waives his proverbial pink hat to salute the crowd. He sits down

and he takes the head in his hands. After three Olympic medals, uncounted championships, thousand of beach volley wins and a two minutes standing ovation he is not happy. He missed the last ball.

Luke started to blog again

Fri, 27 Apr 2007 12:39:37 GMT

After a long hiatus, Luke goes at it again. Luke is our compiler PM and one of the most original coders I’ve seen.

It always makes for a great read: http://blogs.msdn.com/lukeh/default.aspx

A financial education (my ideas)

Fri, 20 Apr 2007 14:55:39 GMT

When I came to US, I wanted to get an introduction on personal finance. I needed to know what to insure, when, how to buy an house and all this sort of basic things. At the time I read Personal Finance for Dummies. I then discovered that very many US folks don’t know about these things either and I found myself recommending this simple book to plenty of people. You can find it for a few bucks in used book-stores. It might be too basic for you, but again, it might not.

Investment is a subset of personal finance and by far not the most important one (the most important probably being how to save the most – which means paying you first). Nevertheless, understanding how to invest the money that you do save is important. The following represents a very condensed summary of my views on the topic. There are dozens of others very good books that I could have suggested instead.

The first thing I would do is to get a decent understanding of the relationship between risk and reward that is so pervasive in financial markets. Good books on the topic are The Four Pillars of Investing or A Random Walk Down Wall Street. A new book that I haven’t read, but suspect quite good (and short) is The Little Book of Common Sense Investing. This books try to convince you that it is not possible to ‘beat the market’ and the best you can do is to decide on an asset allocation that represents your risk/reward profile. If you believe that, then it is worth getting to the next step by understanding the concept of mean variance optimization. Good medium level books on that are All About Asset Allocation and the Intelligent Asset Allocator. There are advanced books on the topic as well, but I won’t mention them here. An advanced service in this area is IndexInvestor.

With time, analyzing the amount of evidence, I came to believe that it is indeed possible to beat the market. But it is not easy and 99% of the methods out there are bunk. Moreover it does require a lot of work and a certain passion for the investment field. So, if you don’t have time or passion you are probably better off sticking to the strategies described in the aforementioned books. If you have a small amount of time, but you still want to try and beat the market, an avenue open to you is the selection of mutual funds. Even then, you need to have some strong held view of which kind of investment strategy and character traits are more likely to bring about superior investment results so that you can pick the right manager (or most likely set of managers) for your money. I describe below what are my personal views on the matter. They are all very debatable. A good service that gives sane information about mutual funds, how to select them and assemble them in your portfolio is Morningstar. I certainly suggest you subscribe to it, if you invest in mutual funds.

A time honored and generally successful stock picking method is value investing. The father of value investing is Benjamin Graham and two books written by him are everlasting classics of investment. They are The Intelligent Investor and Security Analysis. Aficionados disagree on which releases are the best, the links provided represent my opinion. Please start with the former, as the latter is long and complex. They are both quite dated, but plenty of folks believe that not much has been discovered after their publication. I disagree. Another, maybe better, introductory book is The Little Book of Value Investing. There are wonderful mutual funds out there that invest in the spirit of value investing. Among others, I like Third Avenue, Pinnacle and Royce.

Warren Buffett expanded on Graham by putting more emphasis on the quality of the companies he invests in instead of mostly the price. A good book describing his approach is The Warren Buffett Way. You can obviously go and read Buffett’s letters directly here, but that is better suited as a second step after reading Graham or the aforementioned book. There are plenty of disciples of this method of investing. If you want to have some of them managing your money, my preferred are Longleaf, Fairholme, Weitz and Clipper among others. A good service that suggest stocks based on this style of investing is the stock section of Morningstar. You also can invest in brk.b directly to buy the services of Buffett himself. I personally believe it is quite undervalued at the current juncture.

In the past 15 years or so, there has been a lot of interesting work done on purely quantitative investing systems. They have several advantages over judgment based ones (essentially the human brain is not wired correctly to fight a complex adaptive system as the stock market). There are two categories of systems I feel I can recommend in this area: fundamental based and momentum based. This is a very complex mathematical topic at the high end, but it is possible to get a profitable understanding of it by reading WWOWS and The Little Book That Beats The Market. These books describe fundamental based systems, for a purely momentum based one the best bet is FundX. The web site is not too informative, but if you go to the library and ask the librarian they have a little book written by the FundX guys that describe their system in a great level of detail. The best quantitative mutual funds I can think of are from Bridgeway (there are also others). Like all the other funds I suggested, they are managed by wonderful people that takes the shareholders interests very seriously.

You can also try to tactically switch between different asset classes (i.e. stocks and cash) based

on some indicators. I have mixed feelings about this. It might work, but it is debatable how much do you really gain by doing it, after considering taxes and commissions. It is something I could investigate more. Two good books on the matter are The Only Three Questions That Count and The Research Driven Investor. Good mutual funds that do this (with very different risk profiles) are Hussman and CGM.

A couple of notes to close. Reading these books is an exercise in critical thinking. There are huge amount of contradictory statements on very basic and important concepts. The only way around it is to do your own research and make up your own mind. Even then, you won’t be in a situation of perfect information/clear decision. This cannot be so because the market changes constantly. As soon as something is discovered it gets priced in. This is why it is such an exciting field of study (for some people at least). Also I have my own key rule about investing: I never invest in anything that I don’t understand completely. Every time I’ve done that, I’ve regretted it. If this is the only thing that you remember after reading this, I’ll be very happy.

Which type should I use in C# to represent numbers?

Tue, 27 Feb 2007 18:09:57 GMT

Yesterday I found an old email in my mail box that I thought might be generally interesting.

I was asking the technical lead on the C# compiler which algorithm/shortcut people should use to choose their ‘number types’ among the many available in the language. I was asking for something that works the majority of times, even if not always. I’m sure there are other scenarios we haven’t consider. Anyhow, here is his algorithm.

If you need fractions:

Use decimal when intermediate results need to be rounded to fixed precision - this is almost always limited to calculations involving money.
Otherwise use double - you will get the rounding of your calculations wrong, but the extra precision of double will ensure that your results will be good enough.
Only use float if you know you have a space issue, and you know the precision implications. If you don’t have a PhD in numeric computation you don’t qualify.

Otherwise:

Use int whenever your values can fit in an int, even for values which can never be negative. This is so that subtraction operations don’t get you confused.
Use long when your values can’t fit in an int.

Byte, sbyte, short, ushort, uint, and ulong should only ever be used for interop with C code. Otherwise they’re not worth the hassle.

A good Emerging Market fund

Fri, 02 Feb 2007 13:26:00 GMT

If you are looking for a good fund for this corner of your portfolio, you might want to consider Eaton Vance Structured Emerging Mkt (EAEMX). There is also a much cheaper version (EITEX), but it is open just to folks with a financial advisor.

It is a mechanical fund. Here is how their strategy works (as far as I understand it):

They divide EM market countries in 4 categories, they assign a weight to each category (I.E. 20% to tier1, 30% to tier2 etc)
For each category they equal weight each country in the category (I.E. in tier1 equal weight Russia, Mexico, etc)
For each country they equal weight each sector
They actively harvest tax losses

The end result is as follows:

Smaller country have more weight than in a normal Em.Mkt. fund. This is good because:
Smaller markets have had higher returns in the past
Smaller markets are less correlated than large markets both among themselves and against the SP500
Equal weighting the sectors has the following positive effects:
You get more exposure to sectors, like Services, that are going to benefit from economic growth
You avoid the ‘commodity effect’: the tendency of Em.Markets to fall when commodities prices fall because such a large slice of their economy is based on it
It is extremely tax efficient because:
The mechanical strategy lends itself to low turnover
They actively harvest tax losses

In essence they maximize the positive effects of diversification and tax efficiency, which are very stable effects. They are mathematically provable 🙂 The real time results confirm the theory as this has been one of the best performing Em. Markets funds with the lowest standard deviation and best tax efficiency. No-one knows what the future holds, but there is a good chance that this will continue to be the case.

I don’t own it currently because I don’t like the high fees, but the strategy makes so much sense that I might end up buying into it.

Apparently simple code

Thu, 01 Feb 2007 14:54:00 GMT

Sometimes what looks simple is complex and what looks complex is simple. See if you can understand how this one calculates all the possible ways to give change for a certain amount of money given some kinds of coins. You MIT guys out there don’t count, you probably have read the solution in the same book I have.

BTW: the code works with the LINQ May CTP …

using System;

using System.Collections.Generic;

using System.Text;

using System.Query;

using System.Xml.XLinq;
class Program
{
static void Main(string[] args)
{
var coins = new int[] { 1, 5, 10, 25, 50 };

var i = ChangeComb(100, coins);
Console.WriteLine(i);
}

static int ChangeComb(int amount, IEnumerable<int> coins)
{
if (amount == 0) return 1;
if (amount < 0) return 0;
if (coins.Count() == 0) return 0;

return ChangeComb(amount, coins.Skip(1)) +
ChangeComb(amount - coins.First(), coins);
}
}

Is this functional code? Dr.T. Reply.

Wed, 31 Jan 2007 12:34:00 GMT

This would be a functional approach:

CountWithPrevious : [‘a] => (a => a => bool) => int // type of the function
CountWithPrevious [] _                    = 0
CountWithPrevious [_]  _                  = 0
CountWithPrevious [prev, val | tail] pred = CountWithPrevious [val | tail] + (pred val prev ? 1 : 0)

Some observations:

_” is used as wildcard argument – matches any value that you don’t need a name for. [ x,y,z | t ] are pattern matching over lists – here x,y and z get bound to the first elements and t is the rest of the list. Both CountWithPrevious and the passed-in pred are curried – they take two arguments but one at a time. The CountWithPrevious function is defined with pattern matching – at runtime it searches through the definitions until one matches. The type declaration is optional – the compiler can figure out the type from the last case of the function. In C# we don’t have pattern matching and currying, and so would probably need a helper function.

public static int CountWithPrevious<T>(IEnumerable<T> en, PredWithPrevious pred) {
IEnumerator<T> rest = en.GetEnumerator();

if (rest.MoveNext()) return Helper(rest.Current,rest,pred);

else return 0;

}

private static int Helper<T>(T prev, IEnumerator<T> rest, PredWithPrevious pred) {

if (rest.MoveNext()) {

T val = rest.Current;

return Helper(val,rest,pred) + (pred(val,prev) ? 1 : 0);

} else return 0;

}

We could simulate local functions with lambdas so that we don’t need to pass pred, prev and T along:

public static int CountWithPrevious<T>(IEnumerable<T> en, PredWithPrevious pred) {
IEnumerator<T> rest = en.GetEnumerator();

Func<T,int> helper = prev => {

if (rest.MoveNext()) {

T val = rest.Current;

return Helper(val) + (pred(val,prev) ? 1 : 0);

} else return 0;

};

if (rest.MoveNext()) return Helper(rest.Current);

else return 0;

}

Is this functional code?

Fri, 26 Jan 2007 12:54:00 GMT

I’m an Object Oriented (OO) kind of guy, maybe a bigot. I have read a very large number of OO books and I’ve written a fair amount of OO code in my 10+ years in this industry. I’m afraid my mind is wired for OO at this point, for good or bad.

Recently, I’ve been getting interested in functional languages. The most functional guys around here are Wes and Luke, while Dr. T has somehow transcended the whole thing.

Here is the text of one of my newbie emails to Dr.T. I’ll let you think about it for a bit before posting his reply. The phrase: “Is this it?” has to be read as “Is this what functional programming is all about?”.

/// EMAIL TEXT BELOW

Is this it? I have about four of these ‘delegate taking’ functions in my excel addin ….

    delegate bool PredWithPrevious<T>(T value, T previous);
static int CountWithPrevious<T>(IEnumerable<T> en, PredWithPrevious<T> pred) {

int count = 0;
T previous = default(T);
bool isPrevious = false;

foreach (T val in en) {

if (isPrevious)
if (pred(val, previous))
count++;
previous = val;
isPrevious = true;
}
return count;
}

A trading/portfolio management Excel Add-in based on the books by Ralph Vince

Thu, 04 Jan 2007 17:53:00 GMT

I like to write code and I still manage to do it even now that it is not my primary job. I never post the things that I write because I don’t want to maintain them. Lately Charlie convinced me that I don’t have to do that. I can just throw the code out, without much preparation or implicit contract of perpetual maintenance.

This one is an Excel add-in that adds functions to Excel to analyze your trading and manage your portfolio. Notice that I’m not a professional trader or statistician, so the whole thing could be wrong, buggy or conceptually absurd. Probably it is all of the above.

I used the extremely good ExcelDna to write the add-in. You need to download it and follow the instructions in HowToInstall.txt on the attached zip file to use it. I based the formulas mainly on the work of Ralph Vince. Please buy and read his books on money management as they are wonderful.

On the statistics side: I made up the Downside correlation coefficient, I have no idea if it is statistically sound.
On the technical side:

I haven’t optimized the algos at all. I.E. I’m sure there is a way to calculate the standard deviation without navigating the array twice, but I didn’t bother to look. They end up being fast enough anyhow (apart from the Monte Carlo related functions that I should investigate).
I haven’t organized the code correctly. It needs to be rewritten now that I partially know what I’m doing.
I haven’t used LINQ. It will be a lot of fun for me to rewrite the code to be more OO and to use LINQ in the process.

I plan to implement OptimalF for a normal distribution and Efficient Frontier calculations in the future. I have no idea when.

I also have some other projects that might be interesting to share (I.E. a blackjack simulator that you can use to try the result of different strategies inspired by Blackjack Attack).

This is the list of Excel functions that you get with this add-in. They ends up in a Trading category in the list of functions on the Insert Function Excel dialog:

AnnualizeRet(double, double)
AnnualizeStdevp(double, double)
AnnualStdevpParam(double, double)
ArithMean(object[,], double, double, object, double, double)
AutoCorrel(object[,])
AvgLoss(object[,], double, double, object, double, double)
AvgWin(object[,], double, double, object, double, double)
Chebyshev(double)
Correlat(object[,], object[,])
Describe(object)
DownCorrelat(object[,], object[,], double)
DownStdevp(object[,], double)
EGM(double, double)
EGM2(double, double)
GM(object[,], double, double, object, double, double)
Kelly(double, double) TradingLibrary.Losses(object[,])
MaxDD(object[,])
MaxDDFromEq(object[,])
MaxLoss(object[,])
MaxNegRun(object[,])
MaxWin(object[,])
MCMaxDD(object[,], double)
MCOptimalF(object[,], double, double, double)
OptimalF(object[,], double, double)
OptimalFArray(object[,], double, double, double)
OptimalPos(double, double)
PortAnnualGM(double, double, double, double, double, double, double, double)
PortAnnualTaxGM(double, double, double, double, double, double, double)
PortAnnualTaxGMGivenTrades(object[,], double, double, double, double, double, object, double, double)
PortStdevp(double, double, double)
PRR(object[,], double, double, object, double, double)
PRRParam(double, double, double, double)
RealReturn(double, double, double, double)
RunTest(object[,])
SharpeRatio(double, double, double)
SortinoRatio(double, double, double)
Stdevp2(object[,], double, double, object, double, double)
TWR(object[,], double, double, object, double, double)
Wins(object[,])

TradingLibrary.zip

Our sessions at the PDC

Thu, 14 Jul 2005 12:03:00 GMT

Now we can finally blog about our PDC sessions. We have been working for more then a year on this data language integration stuff and it is quite exciting to be able to talk about it publicly. Here are the sessions involved. They are all tied together by a common theme. They are all part of the same big project.

The .NET Language Integrated Query Framework: An Overview’
This is an overview of the overall project. Anders is an excellent speaker and this is a prerequisite for the other sessions. Don’t miss it

C#: Future Directions in Language Innovation from Anders Hejlsberg
Mr. Anders again. If you are a C# programmer, this is one you really want to see

Using the .NET Language Integrated Query Framework with Relational Data
Despite the speaker (probably yours truly) this is a great talk, databases and objects, hmmm sounds interesting

Using the .NET Language Integrated Query Framework with XML Data
Do you want to see how to query XML in a rather innovative way? Here you have it

Visual Basic: Future Directions in Language Innovation
This is the equivalent of Anders’ C# talk. Come and see where VB is going

There is a C# Language chat coming

Thu, 14 Apr 2005 18:40:00 GMT

Here it is: http://blogs.msdn.com/scottno/archive/2005/04/14/408212.aspx

Here is a message board about the C# language

Mon, 28 Mar 2005 11:06:00 GMT

Here is the link: http://www.gotdotnet.com/Community/MessageBoard/MessageBoard.aspx?ID=5627. Check it out.

To rent or to buy a house, some more evidence of my unpopular thesis

Wed, 09 Mar 2005 12:01:00 GMT

I discussed this topic in a previous post http://blogs.msdn.com/lucabol/archive/2004/07/30/202394.aspx and I received a number of emails telling me how crazy I was even speculating that, at the current juncture, it may be more convinient to rent than to buy a house.

This week The Economist runs a piece where they make the same exact claim. House prices are so high and rents are so low all over the world that renting might, indeed, be better. Even considering the low interest rate environment, it would seem that the house prices are out of step with reality, very much as equities where in the 90’s.

It is all a game of expectations. The harder things to estimate are the future house price gain and the interest rate/gain you can get by investing the difference between rent and mortgage payments. By using difference estimates, you get difference answers. My own opinion is that the probabilities weight against buying an house at this time. Here is a spreadsheet to help you out: http://blogs.msdn.com/lucabol/archive/2004/08/18/216662.aspx .

There are other considerations: I bought a house not to get killed by my wife 🙂 But I’ve been able to limit the expense and I avoided investing more money in real estate.

Compiler trivia: const, operators and being nice to the compiler

Thu, 10 Feb 2005 10:32:00 GMT

This is a question that came up on our internal alias. I thought it might be generally interesting to illustrate how the compiler picks operators.

Here is the original issue. This code compiles fine:

UInt64 vUInt641 = UInt64.MaxValue;
const int vInt2 = 1432765098;
int res = (int)(vUInt641 - vInt2);

But this code generates a compile error:

UInt64 vUInt641 = UInt64.MaxValue;
int vInt2 = 1432765098;
int res = (int)(vUInt641 - vInt2);

(line 3): error CS0019: Operator ‘-‘ cannot be applied to operands of type ‘ulong’ and ‘int’

The only difference between the two pieces of code is the presence of the const keyword in the first one. Let’s first analyze the second case. The reason an error is generated is that there is no ‘-‘ operator defined between an ulong and an int. There is also no implicit conversion between int and and ulong or the other way around. The compiler has to give up and to produce an error.

In the first case the variable is marked as const, which means that the compiler knows its value at compile time. It realizes that the value is positive and can safely been converted to an ulong. The compiler converts it and then invokes the -(ulong, ulong) operator.

A bizarre way to think of it is this. As you have been nice to the compiler by telling him that you are not going to modify this value, the compiler then is nice to you by making use of the info to help you out in this case

Remember, always be nice to the compiler, the more you tell him, the more he tells you

Nullable trilogy part III: Nullable as type parameter in a generic class

Thu, 03 Feb 2005 15:42:00 GMT

Another commonly asked question relates to the behavior of Nullable when used as type parameter to instantiate a generic class. It might be surprising that comparing such a parameter to null gives always false as a result. As it turns out, this is not related to Nullable, but it is a result of how generics are implemented. There is a tendency to think about generics in a manner very similar to C++ templates, this view is unfortunately not correct.

Generics have a runtime representation and, as such, the compiler needs to generate IL that works whatever type is used at runtime to instantiate the generic class. This means that the compiler cannot call user defined operators, because at compile time it has no knowledge about them. The compiler doesn’t know at that point which type will be used to instantiate the generic class in the future. It cannot know that it has to generate code to call these user-defined operators.

In the same vein the compiler doesn’t know that a generic class will be instantiated with a Nullable parameter and so it cannot produce IL to lift operators (i.e. the equality operator ‘==’ ). The result is that when the programmer writes code like ‘t==null’, the compiler generates IL to call the ‘standard’ ‘==’ operator, which in the case of Nullable returns false because t has the runtime type of struct.

A similar behavior is observable with the following code using strings:

string s1 = Bob John;

string s2 = Bob;

Console.WriteLine(Equals(s1, s2 + John));

static bool Equals(T a, T b) where T:class {

return a == b;

}

This code would return false, because the ‘==’ operator for reference types gets invoked.

A case could be made that the compiler should generate IL to check the runtime type of the generic parameter and call the correct operator for at least some well-known types. This solution wouldn’t work for user defined operators as the compiler doesn’t know at compile time the set of types that could be used to instantiate a generic class. In general we don’t like to keep this sort of ‘lists of special things’ in the codebase, More importantly the solution would impose a somehow significant performance penalty at each use of the operator. This feeling of ‘hacking’ things and the significant performance problem convinced us not to implement this solution.

Nullable trilogy part II: a == b -> a>=b && a

Thu, 27 Jan 2005 07:54:00 GMT

A question that often comes up when we discuss Nullable is about the anti-symmetric property. This property states that if a==b then a>=b and a<=b. If a and b are null then this property is not satisfied in the current design as the result of >= and <= is always false when one of the parameters is null. This may seems surprising, but it can be easily understood when considering if null can be ordered.

We decided that null being less of (or more of) all the other elements in the domain would be an arbitrary decision and as such null cannot be ordered. If it cannot be ordered then the result of the relational operators for null is false as these operators can be defined as:

  a<b -> ordered(a,b) && a<b

  a<=b -> ordered(a,b) && (a<b || a==b)

It might be argued that for practical reasons it is convenient to have the <= and >= operators return true in case null is on both sides, but this choice has some practical drawbacks. Consider the following code:

    void ProcessTransactions(int?[] transactions, int? maxValue) {
    foreach(int? t in transactions)
      if(t < maxValue) ProcessTransaction(t);
    }

This code does what the programmer expects, even when maxValue is null (in which case it won’t process any element). Now let’s suppose that the programmer changes his mind and wants to include maxValue in the processing. He will then change the ‘<’ operator to be ‘<=’.

Under the current design he will obtain what he expects, even in the case maxValue is null. By considering null >= null as true, this function would suddenly start processing all the null transactions, which is not what the programmer intended. The problem here is that people tend to think about the ‘>=’ and ‘<=’ operators as a way to include the limit value in the set of things to be processed. Very rarely, they intend to process the null value as well. Another way to think about it is that: deciding that null cannot be ordered implies that >= has to return false to be conceptually consistent and to prevent these quite subtle bugs to occur.

New C# things happening

Mon, 24 Jan 2005 09:49:00 GMT

Express yourself about this DCR: http://blogs.msdn.com/scottno/archive/2005/01/19/356347.aspx

Partecipate to the C# language chat: http://blogs.msdn.com/scottno/archive/2005/01/19/356358.aspx

That's it.

Nullable trilogy Part I: why not just SQL?

Thu, 20 Jan 2005 09:41:00 GMT

This is the first of a weekly three part serie of posts about Nullable. In these posts I want to describe the reasons behind three design choices:
1. Why not just use SQL semantics for null?
2. Why null == null doesn’t imply null >= null and null <= null?
3. Why inside a generic class with a type parameter t the expression t == null will return false, when t is a nullable type and the value of it is null.

Let’s start from the first question as the answer is shorter. We’ll get to the other two in the coming weeks.

The first question relates to the reason not to have the same semantics as SQL for relational operators. The SQL semantics have been commonly referred to as three-value logic where null == null returns null. Introducing such logic in the C# language would be problematic. The main reason is that the language already contains the concept of null for reference types and it does have the programming languages traditional two-value logic where null == null returns true.

Granted that we cannot change this definition, then the addition of three-value logic just for some types would be confusing. We would need, for example, to create a new NullableString class to be able to apply three-value logic operators to it. More generally, the presence in the same code of two value logic and three value logic operators would make the code quite difficult to write, read and maintain.

x+=x++;

Tue, 31 Aug 2004 15:00:00 GMT

Today I was looking through an internal C# user group list. One of the messages was about x+=x++; and what should be the right behavior. I thought this is more generally interesting, so I decided to blog it. But first, DO NOT WRITE THAT CODE!!!

Ok, with that out of the way, we can start…

Let’s take this code as an example:

int x = 3;

x += x++;

The first thing the compiler does whenever it sees something like z += y is to convert it to z = z + y. This is obviously true for +=, -=, *=, and /=. Ok, this was easy. Now we have just to consider:

x= x + x++;

This, by the way, gives the same result as:

x = x + x;

This, by the way, gives a different result from:

x = x++ + x;

This, by the way gives the same result as:

x = x + ++x;

As maddening as this may seem, it actually makes sense (once you understand how it works). But first, what is the difference between x++ and ++x? x++ returns the value of x to the current expression and then increments x. ++x increments x and then return its value to the current expression. Given this factoid (and knowing that c# evaluates expressions left to right), we can then consider what happens in the following case:

int x = 3;

x = x + x++;

Here is how the compiler conceptually evaluates it:

x = (x) + x++ -> the first x gets evaluated and returns 3, x = 3
x = 3 + (x)++ -> x gets evaluated and returns 3, x = 3
x = 3 + (x++) -> x++ gets evaluated and x is incremented (to 4), x = 4
x = (3 + 3) -> 3 + 3 gets evaluated and returns 6, x = 4
(x = 6) -> x is assigned to 6 (overriding the previous value of 4)

Now let’s see how this one works:

int x = 3;

x = x++ + x;

x = (x)++ + x -> x gets evaluated and returns 3, x =3
x = (x++) + x -> x++ gets evaluated and x is incremented, x=4
x = 3 + (x) -> x gets evaluated and returns 4, x = 4
x = 3 + 4 -> 3+4 gets evaluated and returns 7, x = 4
(x=7) -> x is assigned to 7 (overriding the previous value of 4)

Now let’s get to this one:

int x = 3;

x = x + ++x;

x = (x) + ++x -> x gets evaluated and returns 3, x=3
x = 3 + (++x) -> ++x gets evaluated and x is incremented, x=4
x = 3 + (x) -> x gets evaluated and returns 4, x=4
x = 3 + 4 -> 3+4 gets evaluated and returns 7, x = 4
(x=7) -> x is assigned to 7 (overriding the previous value of 4)

I hope this is clear. By the way, in c++ the behavior for this expression is undefined…

But now… why did we make this legal? Why not err or warn at compilation time? Well…

We were wrong, we should have erred or warned, but now it is too late because if we change this we break code OR
It is quite complex to form a set of guidelines that the compiler can evaluate to be able to err just in the bizarre cases OR
We prefer to spend our time working on things people really care about instead of these corner-corner-corner cases

Does it matter which of the previous options is the correct one? Not really because…

YOU ARE NOT GOING TO WRITE THAT CODE J

Homeowner association transcripts

Wed, 18 Aug 2004 12:19:00 GMT

Does anyone know if the homeowner association president for a condominium complex is legally forced to give the transcripts of their meetings to prospective buyers?

It is quite funny, the lady in charge just doesn’t want to talk to me. I know, I know, I shouldn’t buy that condo then love over reason

Rent or buy spreadsheet

Wed, 18 Aug 2004 12:13:00 GMT

Someone asked me for the spreadsheet I used for the calculations in my previous posts. Here you have it. The data doesn’t reflect my current situation.

Feel free to use it with your own numbers (and tell me of any bug).

How much object relational framework do you really need?

Fri, 06 Aug 2004 09:05:00 GMT

It is always interesting for me to look at debates about object relational layers. I propose we define different levels of object relational support:

No support: all the objects are persisted and queried by writing ADO.NET code by hand
Code generation: you run a tool on your database (or on an abstract description of your schema) and the tool generates a bunch of objects for your tables and some ADO.NET code to retrieve/persist their state
Total code gen: the tool generates all the ADO.NET code
Partial code gen: the tool generates just code to intercept property accessors and provide delay loading, but all the data access code is in a separate compiled component
Metadata based: no code generation phase. The access to fields and property goes through reflection and the data access code is not exposed in the user code
Semi transparent: the persistent classes or properties need to be marked in some special way to be persisted (attributes, inherit from a special class or such)
Transparent: the classes don’t need to be modified at all to be persisted

Things get fuzzy, though. It is sometime unclear the difference between 2.2 and 3.1 and various creative solutions can be hard to classify. But in a general sense, this classification is probably about right. In a generic sense EJB1.1-2.0 is a 2.2, EJB3.0 is a 3.1, JDO is 3.2 (if you don’t consider post-compilation) and ObjectSpaces is a 3.2. Hibernate and Toplink are squarely 3.2.

But do you really need to go all the way to 3.2? All the times? I’ll try to post more about trade-offs in all these solutions, but if you have an idea of a better categorization, please let me know. The one I propose is right out of my head and I’m not to happy about it either.

I am really an old, almost fat, beach volley player

Thu, 05 Aug 2004 08:37:00 GMT

I used to play a lot, back in Italy. We would come out from work, go to the beach, play two or three hours, swim a bit and then go out for the night. This would happen about every day. We took it awfully seriously. We would go to the Italian national championships and all. It was quite a big deal.

Seattle, on the other hand, is not too conductive to beach volley. Not too many real beaches and all of them quite far (commute wise) from where I live. So I don’t play anymore, but I still like the game. One weekend or two each year, my wife and I go down to Los Angeles to watch AVP tournaments. And man, it is a lot of fun!!

And, by the way, beach volley has a lot going for it. For the players it is a low impact sport (the sand is soft). Plus you play the ball a lot, much more than in normal volleyball (or even soccer). Your are always in to the game, no place for hiding. It happens under the sun, so you don’t get all the nasty problem of not warming up correctly. And the environment oh man!! You are on the beach, not in some crazy smelling gym somewhere. All in all, you got to love it.

Anyway, down memory lane

Book: Objects, Components, and Frameworks with UML, Fedmond F. D`Souza

Tue, 03 Aug 2004 13:13:00 GMT

This is a book about Catalysis, an OO methodology. I read this book some time ago (like years). It is a quite complex book. If you don’t like methodologists, it will easily put you to sleep. But if you stay awake, it is worth it.

It is strange how this book crystallized one concept for me: the definition of an interface. An interface is not just the sum of the signatures of the methods. It is a conceptual framework. The only way to fully describe it is to describe in a formal way this conceptual framework. For example without further specification it is impossible to know how to use this interface:

interface IBook() {

void AddPage(IPage p);

}

What happens when you add a page? Will it add it at the end of the book? At the start? What happens if you call the method twice? Will the pages be in order? Will they be in the same section? Is there a concept of a section?

There are ways to make explicit the implicit model beyond an interface: the one that I’ve commonly seen being used is documentation. The book argues for formal contracts (preconditions, postconditions, invariants). The latter is certainly more formal, but to use it you need to describe formally what is the state beyond the interface and how the different methods affect this state. It is a lot of work.

Is it worth the effort? I guess it depends on a zillion things like: the type of project, the people involved, I don’t think anybody is actually doing it, but it is still a good conceptual framework to keep in mind when you define your interfaces: it is not enough to write down the signature methods, you have somehow to expose the conceptual framework (where somehow means more or less formally).

I am buying a house

Mon, 02 Aug 2004 12:57:00 GMT

I think I’ve been misinterpreted in my previous posts. I’m not suggesting that renting is better than owning. You have to run your numbers and make your own choice. There are many assumptions to consider (financial and not). The not financial ones often are more important. You have to run your numbers, but in general renting looks better if:

You cannot buy ( 🙂 ) and/or
You plan to stay in a place for a short time and/or
You believe houses are vastly overpriced in your area and they’ll return will be negative for a while and/or
You don’t want the psychological part of having a house (maintenance, tax payments,)

In my case the most worrisome scenario is 3.

Back to our situation, we are buying a house for the pleasure of living in a particular location. Financially it is an OK transaction (fair price, I think).

I’ll probably post a little rent-buy spreadsheet as soon as I have time to refine it.

Thanks for all the good comments.

To rent or to buy a house??

Fri, 30 Jul 2004 11:06:00 GMT

Spurred by some comments to a previous post I quickly run these numbers. They analyze the case of house price not rising in the next five years. I'll take more time to check them if I really get to a point of deciding, but they should be correct.

This data (fairly typical in my area):

Home price: $200,000

Down payment: $0

Interest rate: 6%

Monthly mortgage: 1210

Rent: 750

Annual gains on investments: 6%

If I run this for 5 years and house appreciation 0% I get:

$33,043 on investing the rent-mortgage difference

$30,609 on paying the mortgage (equity accumulation + tax savings at 28%)

If we assume to put down 20% to buy the house then the numbers are even worse:

$83,542 on investing the down payment and the rent-mortgage difference

$67,548 on paying the mortgage (equity accumulation + tax savings at 28%)

There are other things to consider: house repairs for houses, annual raises in rent, etc etc

Are those assumptions reasonable? So it is better to rent than to buy? Well, over the long term houses do tend to appreciate and the more leverage you have, the more you reap the benefits. Over the short term, run your numbers. There are so many assumptions you can make, that it become almost a personal opinion what is best.

Running the numbers help you in making sure than it is at least an informed opinion. It makes your assumptions explicit

My quest for a new home

Thu, 29 Jul 2004 09:10:00 GMT

I'm refining my understanding of how to value residential properties. There is this rule of thumb: buy when you can get a rent for the property which is at least 1.2% of the property price (or in another form, when the home price is 7x the annual rent). This is an approx for a more formal spreadsheet (which I have).

This is fine and dandy, but in the area I'm interested in a typical one bedroom condo sells for around 200,000-300,000 and you can get a monthly rent of $1,000-1,300. This doesn't work by a very large factor. Hmmm

Let's change the target to a cheaper area. Now I can probably get around 800 monthly. To make things work I need to find a condo for $67,200. Not an easy task, almost impossible.

It looks like for me it is hard to find a house in an area I like and at the same time make a good investment (unless the market bails me out by raising home prices at a high rate). There is quite a large disconnect between house prices and rents now. Maybe the residential market is really on the brink of a fall Time to rent again?

Book: The coming crash in the housing market, John R. Talbott

Wed, 28 Jul 2004 08:56:00 GMT

My wife and I are planning to move somewhere where you can walk to shops and restaurants. We were thinking about selling or renting our current suburbia home. So, as usual, I picked up a bunch of books from the local library. This is one of them. Amazon reviews are here.

My opinion on the book

The author did a good job of providing numeric evidence of the possibility of a real estate bubble. What I didn't like is that he didn't provide a logical framework to evaluate residential real estate. I was expecting some sort of DCF model that uses rent as the free cash flow. Instead he cites Median house/Median Income and some other contingent evidence to prove his point. It is like a book on stock evaluation that just uses historical P/E as a measure. He also fail to present similar situations that resulted in a crisis. Quite disappointing in that regard

How it relates to my choice of sell or rent

I created my own excel spreadsheet to look at the two different scenarios (if you are interested I can share). It looks like renting is the loosing one. We gain more by selling and investing the equity in a conservative way. The book made me take a hard look at our local real estate market: the ratio Median House/Median Income is about 4.3 (above 3 is bad). Rents are low and house prices have been growing like crazy. This makes me think to buy a small condo, or even to rent for a while. We haven't really decided yet.

Book summary

1. House prices are high while everything else is low: rents, economy, stocks Yes, there are bigger houses now and rates are low (but won't stay low forever)

2. Mortgage takes more and more of the average salary, foreclosures and personal bankruptcies are at record high. People is more and more leveraged

3. The house market is far away from an ideal market. A bunch of lenders really make the prices. Bubble can easily create in inefficient markets

4. People think houses can just go up: same mindset as the internet bubble

5. House prices have been growing like crazy. The fast growers of the past are the fastest growers now: it is like momentum in stocks

6. Many of the previous crashes were based on leverage and some kind of Ponzi scheme

7. It will start if rates goes up (people can afford less), if they go down (the economy will be very weak), banks become less aggressive in lending, congress reduces tax benefits

8. Under a worst case scenario Fannie Mae and Freddie Mac ($3 trillion of very leveraged and derivative invested money) stumble and then it may be a worldwide depression as banks credibility goes down

9. What you can do: decrease your exposure to real estate (rent, downsize or move to cheaper area), be less leveraged, hedge your real estate exposure by not buying real estate cyclical stocks

Please the perfect object stand up!!

Tue, 27 Jul 2004 15:21:00 GMT

I always thought: why people write this very long posts? Here you have it.

How do I design my objects? What is the set of constraints that dictate my design decisions? People talk about objects in very different contexts. A certain amount of confusion derives from not identifying the contexts for our objects.

First, kings of them all, there are business objects. My library is full of OO books that describe how to design these guys. We have patterns and anti-patterns. Everybody is out there promoting their own methodology. But if we go deep enough, the most important design task in creating them is assigning responsibilities. This concept is as old as OO itself; it finds its roots in CRC cards. But I digress. Once again, when you design your business objects you have to be very careful to assign responsibilities to the objects that have the right information to perform them.

It would be nice if this was all you have to code. A crazy bigot (like me) would think that all the rest is infrastructure (ala naked objects). But we live in the real world, so let's get down to it.

A second category of objects I often encounter are network objects. These are the guys you use to communicate data across tiers (logical, physical, webservices). They are usually quite dumb objects, not too much logic in them (if any), and your application creates them by filling their properties from the more noble business objects. Something along the line of the value object pattern. Is assigning responsibilities a very important task here? Heck no!! They don't even have methods!! The most important characteristic they have is bandwidth optimization; you want to retrieve a whole bunch of stuff in one single round trip. More often than not you sacrifice everything else (manageability, good design) to achieve that.

This is why I don't believe the quite common ideal of remoting your business objects. The set of constraints you use to design them is completely different (often opposite) to the constraints needed to go over the wire. When anything goes over the wire it becomes just data. Moreover not all of your business state needs to go on the network, but this is another story

Another common object category is database objects. There is a big debate if they are the same as business objects or not. I dont' want to go there, but If you want them to be the same you usually pragmatically sacrifice a bit of OO purity to make them more similar to the database tables (you don't want to fight your object relational mapping layer, you want to help it out). To say this in a more positive way, you design your domain model with an eye on how you are going to persist it. If, on the other end, you are willing to create a separate set of objects to represent data coming from the database, then the main constraint for these guys is the same as your database tables: normalization. This is not surprising as they are mirrors of your database. Again assigning responsibilities and being network friendly take the back seat in this case.

There are many other object categories I often found useful (i.e. façade objects). I don't want to discuss all of them here. Things are already boring as they are. The central point is that it is very hard to define generic guidelines that work well in all the different contexts. You got to think about your context first.

The funny thing is that I work in a place where we are supposed to design features that are generically interesting in all these scenarios. You use C# to code all of them. Oh well, design is the art of choosing trade offs

My first (and last?) post

Tue, 27 Jul 2004 14:18:00 GMT

Hello folks,

My crazy coworkers finally convinced me to get my own blog. They think I'll say something funny, but I won't - so don't link to me - it's probably not worth it. For the few perseverant ones, here is what I plan to talk about: object orientation, c#, sports, financial stuff and books (not in that order). See you later.

BTW: strange how this blog thing reduces the complexity of one's personality to a bunch of categories on the left of the screen

Here I am.

Tue, 27 Jul 2004 13:59:00 GMT

I’m Luca Bolognese.

I’m the Principal Group Program Manager for the C#, VB and F# languages, compilers and IDEs. Before that, I was the LINQ guy. Before that, I was the ObjectSpaces guy.

In my spare time, I read a lot of books. I mean: a lot. I’m not sure how much I retain. I also try to keep some form of physical shape by playing sports and walking.

All in all, I try to recreate my own Italian lifestyle in the Kirkland area.