Archive

Archive for the ‘Uncategorized’ Category

Powershell package management frustration

Nowadays, powershell comes with commands for package management which may act as a wrapper for nuget and other package management tools. However, it seems mysteriously broken when I try to download packages from nuget.

My assumption was that, installing the nuget ‘paket’ should be as simple as:

install-package Paket

When I try that I get back an error.

install-package : No match was found for the specified search criteria and package name ‘paket’.  Try Get-PackageSource to see all available registered package sources.

If I add the verbose flag it tells me that it search nuget for a package with id ‘paket’ but nothing was found.

VERBOSE: Searching repository ‘https://api.nuget.org/v3/index.json/FindPackagesById()?id=’paket” for ”.
VERBOSE: Searching repository ‘https://www.powershellgallery.com/api/v2/FindPackagesById()?id=’paket” for ”.

VERBOSE: Total package yield:’0’ for the specified package ‘paket’.

A bit desperate I added nuget v2 as an explicit package source instead

install-package Paket -verbose -source https://www.nuget.org/api/v2 -Destination .

And that succeeds. Happy days. I should then be able to register v2 as another package source and forget about this.

Register-PackageSource -Name nugetv2 -ProviderName NuGet -Location https://www.nuget.org/api/api/v2/

The strange thing is that it does not help. Running install-package again (without explicit source) clearly states in the verbose output that the v2 source is searched but nothing is found.

Right now, I am confused and frustrated.

Advertisements
Categories: Uncategorized

Package your powershell code

Powershell files

Probably the most natural way to store a couple of lines of powershell script code is to create a file with the extension .ps1. A .ps1 file can be thought of in two ways

  • A script containing utility functions and variables. Such a file can be loaded into your current session using dot sourcing.
  • A function named as the script file and with a function body defined within the script. Parameters must be defined using a param(…) block.

Powershell script modules

Sometimes there is a need to deliver powershell code to a third party; maybe another developer team, the powershell community,  a company or such. Maybe there is a group of related functions and script files that fits naturally together. A powershell module might be the right thing in this case. It is a bit more heavy weight than just a simple script file but, in the simplest form it is still simple – basically you can rename your .ps1 file to .psm1 and you have a powershell module.

I mentioned that there might be several script files that should be grouped together; how is that done?  The most commonly used technique seems to be to simply dotsource the script files into the module file. Some people use snippets like below to batch import a selection of files.

$scripts = Get-ChildItem $PSScriptRoot\scripts -Filter "*.ps1"

foreach ($script in $scripts){
. $script.FullName
}

 

Categories: Uncategorized

Code kata – a final note

The last two posts discussed how to solve the first problem out of five provided on this page. For completeness, I provide my answers to the other problems as well. The code is more standard functional programming (where haskell shines ofcourse but it leaves no surprises I guess).

Categories: Uncategorized

Some code kata – haskell while loops

In the previous post we solved the following problem using a for loop

Write three functions that compute the sum of the numbers in a given list using a for-loop, a while-loop, and recursion.

I have not found while loop primitives in the standard library but a library for this exists on hackage and it is called Control.Monad.Loops. It provides the following implementation

whileM_ :: (Monad m) => m Bool -> m a -> m ()
whileM_ p f = go
  where go = do
           x <- p
           if x
           then f >> go
           else return ()

Looking at the type signature we see that it takes a bool wrapped in a monad as a first argument and another monad which is the action as a second argument. The result, as with the for loop, is a monad containing nothing (or more correct the unit value).

Again, since we want to do stateful programming we will turn to the State monad. This time we need to keep both the remaining list and the sum within the state. A tuple seems like a natural choice.

The most complicated part is the condition. We use the get function to retrieve the current state. The get function has the following type signature

get :: MonadState s m => m s

It simply returns the state as a value wrapped within the monad. However, we want to test wether the list is empty or not. fmap or the equivalent applicative functor <$> is a function that allow us to apply an ordinary function to the value wrapped within the monad – transforming it to something else which in this case should be a boolean value indicating whether we should continue looping or not.

This is the type signature for fmap and <$> respectivly

(<$>) :: Functor f => (a -> b) -> f a -> f b
fmap :: Functor f => (a -> b) -> f a -> f b

To test whether the list is empty we can then do the following (assuming that the list is stored as the first element in the state tuple)

not . null . fst <$> get

The final result, when putting this together, may look like below

import Control.Monad
import Control.Monad.State

whileM_ p f = go
  where go = do
    x <- p
    if x
    then f >> go
    else return ()

sumUsingWhile xs = snd $ execState loop (xs,0)
  where
    loop = whileM_ (not . null . fst <$> get) $
             modify $ \(x:xs,sum) -> (xs, x+sum)

 

Categories: Uncategorized

Some code kata – haskell for loops

First of all – read and enjoy solving these small puzzle problems.

I also did that but with a little twist – I picked haskell which makes the first problem a bit more challenging. Why? Because for and while loops requires mutable states which is something that is not supported by haskell. Unless… you turn to monads and more specifically the state monad.

Lets repeat problem 1:

Write three functions that compute the sum of the numbers in a given list using a for-loop, a while-loop, and recursion.

In the standard library Control.Monad we can actually find primitives for for-loops. The one that I used is called forM_ and has the type signature

forM_ :: (Foldable t, Monad m) => t a -> (a -> m b) -> m ()

The first argument is a foldable type t which is similiar to enumerable in .net. It is generic so any item of type a might be enumerated.

The second argument is the action that we want to repeatadly invoke. It takes as an argument an element of the enumerable. It returns a monad containing a value of some type b. Do not worry too much about what a monad is right now just hang on.

The result is a monad without a value (or for purists: the only value of the unit type). This indicates that the result of the forM_ function is only side effects. Side effects within the monad that is.

The standard monad for side effects is the State monad. To execute a state monad and actually get a result the following function is provided

execState :: State s a -> s -> s

It takes a state monad and an initial state and returns the resulting state.

Last, but not least, we need to be able to modify the state.

modify :: MonadState s m => (s -> s) -> m ()

Using this function we can pass a function that takes the current state as input and returns the new state.

Putting the pieces together we get

import Control.Monad
import Control.Monad.State

sumUsingFor xs = execState loop 0
    where
        loop = forM_ xs $ \x -> modify (+x)

I think that is pretty impressive for a language that does not support mutable states.

Next time, we will have a look at the while loop implementation.

Categories: Uncategorized

Binomial heaps and Purely functional data structures

I had some time lately to contemplate Chris Okasakis book Purely functional data structures. It is a book that requires time. But it pays off. To make sure I understood binomial heaps I made two quick implementations:

Categories: Uncategorized

A first look at elasticsearch graph

September 24, 2016 Leave a comment

Today I sat down to have a look at elasticsearch graph. I needed a dataset to analyze and I picked the enron mail dataset which was released after the enron scandal.

The dataset is provided as a sql script for mysql. I was too lazy to install mysql so I just did a quick and dirty powershell hack to convert some of the data into a csv file.

$lines = Get-Content .\dataset.csv
$lines = $lines | ? { $_ -match "^\s*\(\d+" }
$lines = $lines | % { $_.Trim().TrimStart('(').TrimEnd('),').Replace("`"","").Replace("'","`"") }
Set-Content .\dataset.csv -Value $lines

After this little hack it was simple enough to parse the data using the built-in Import-Csv command.

$header = "mid", "sender", "date", "message_id", "subject", "body", "folder"
$dataset = Import-CSV -Delimiter ',' -Path .\dataset.csv -Header $header

The next thing was to create an elasticsearch index. After some experimentation I came up with this definition.

$mappingDef = @{
  mappings= @{
    email=@{
      properties=@{
        mid=@{ type="string" }
        sender=@{ type="string"; index="not_analyzed" }
        date=@{ type="date"; format= "yyyy-MM-dd HH:mm:ss" }
        message_id=@{ type="string"; index="not_analyzed" }
        subject=@{ type="string" }
        body=@{ type="string" }
        folder=@{ type="string"; index="not_analyzed" }
      }
    }
  }
} | ConvertTo-Json -Depth 10

Invoke-RestMethod -Method PUT "http://localhost:9200/enron" -ContentType "application/json" -Body $mappingDef

 

The next thing is to send all the data to elasticsearch:

foreach($mail in $dataset)
{
  $body = $mail | ConvertTo-Json
  Invoke-RestMethod -Method Put "http://localhost:9200/enron/email/$($mail.mid)" -ContentType 'application/json' -Body $body
}

Assuming elasticsearch and kibana is available, graph is easy to install by following the instructions on the elasticsearch page. Firing up the graph plugin page I was now able to create my first graph:

kibana-graph-sample

Categories: Uncategorized