Giter VIP home page Giter VIP logo

pslitedb's Introduction

🚀 PSLiteDB

Pester Tests

GitHub Workflow - CI

💥 OverView

LiteDB is a noSQL singlefile datastore just like SQLite. PSLiteDB is a PowerShell wrapper for LiteDB

Note: in V5 everything is case in-sensitive

  • Collection names are case-insensitive
  • FieldNames or property names are case-insensitive
  • FieldValues or property values are case-insensitive.

🌀 Clone Module

#Clone the repo in c:\temp
cd c:\temp
git clone https://github.com/v2kiran/PSLiteDB.git

💧 Import Module

Import-Module c:\temp\PSLiteDB\module\PSLiteDB.psd1 -verbose

🌴 Create Database

$dbPath = "C:\temp\LiteDB\Movie.db"
New-LiteDBDatabase -Path $dbPath -Verbose

🎷 Connect Database

  • The connection to the first db is stored in a session variable called $LiteDBPSConnection.
  • This connection variable is re-used in various cmdlets from this module making it efficient by having to type less.
  • if you want to work with multiple databases then you will need to store each connection from Open-LiteDBConnection in a different variable and pass that to each cmdlet's Connection parameter. Check the Wiki for an example on how to work with multiple databases simultaneously.
Open-LiteDBConnection -Database $dbPath

🎺 Create a Collection

New-LiteDBCollection -Collection movies

# verify that the collection was created
Get-LiteDBCollectionName

🎸 Create an Index

# Creates an index in the collection `movies` with all `DisplayName` property values in `lowercase`
New-LiteDBIndex -Collection movies -Field Genre -Expression "LOWER($.Genre)"

# verify that the index was created
Get-LiteDBIndex -Collection movies

🎉 Insert Records

Create a movie database from csv records/file. Force the MovieID property to become the _id property in the LiteDB collection Serialize the selected records and finally insert them into the movies

1️⃣▶️ Insert by ID
# Sample dataset taken from [data-world](https://data.world/)
$dataset = @"
MovieID,Title,MPAA,Budget,Gross,Release Date,Genre,Runtime,Rating,RatingCount
1,Look Who's Talking,PG-13,7500000,296000000,1989-10-12,Romance,93,5.9,73638
2,Driving Miss Daisy,PG,7500000,145793296,1989-12-13,Comedy,99,7.4,91075
3,Turner & Hooch,PG,13000000,71079915,1989-07-28,Crime,100,7.2,91415
4,Born on the Fourth of July,R,14000000,161001698,1989-12-20,War,145,7.2,91415
5,Field of Dreams,PG,15000000,84431625,1989-04-21,Drama,107,7.5,101702
6,Uncle Buck,PG,15000000,79258538,1989-08-16,Family,100,7,77659
7,When Harry Met Sally...,R,16000000,92800000,1989-07-21,Romance,96,7.6,180871
8,Dead Poets Society,PG,16400000,235860116,1989-06-02,Drama,129,8.1,382002
9,Parenthood,PG-13,20000000,126297830,1989-07-31,Comedy,124,7,41866
10,Lethal Weapon 2,R,25000000,227853986,1989-07-07,Comedy,114,7.2,151737
"@

# Convert the dataset from CSV to PSObjects
# Since a csv converts all object properties to string we need to convert those string properties back to their original type
$movies = $dataset |
    ConvertFrom-Csv |
    Select-Object @{Name = "_id"; E = { [int]$_.MovieID } },
    @{Name = "Budget"; E = { [int64]$_.Budget } },
    @{Name = "Gross"; E = { [int64]$_.Gross } },
    @{Name = "ReleaseDate"; E = { [datetime]$_.'Release Date' } },
    @{Name = "RunTime"; E = { [int]$_.Runtime } },
    @{Name = "Rating"; E = { [Double]$_.Rating } },
    @{Name = "RatingCount"; E = { [int64]$_.RatingCount } },
    Title, MPAA

$movies |
    ConvertTo-LiteDbBSON |
    Add-LiteDBDocument -Collection movies
2️⃣▶️ Bulk Insert
$movie_array = $movies | ConvertTo-LiteDbBSON -as array

Add-LiteDBDocument 'movies' -BsonDocumentArray $movie_array -BatchSize 1000 -BulkInsert

👉 Note: The ConvertTo-LiteDbBSON Function returns a Bsondocument array which will be unrolled by the Add-LitedbDocument cmdlet by default so if you want to avoid that and add the entire array. Usefull when inserting a huge number of documents in batches.


❄️ Find Records

Because we used the MovieID property of the dataset as our _id in the LiteDb collection, we can search for records using the MovieID value

1️⃣▶️ Find by ID
#Note that the value of parameter ID: 'BITS' is case-Insensitive
Find-LiteDBDocument -Collection movies -ID 10

Output:

Collection  : movies
_id         : 10
Title       : Lethal Weapon 2
ReleaseDate : 7/7/1989 12:00:00 AM
Budget      : 25000000
Rating      : 7.2
RatingCount : 151737
Gross       : 227853986
MPAA        : R
RunTime     : 114


<#
List all documents in a collection, limiting the total docs displayed to 5 and skipping the first 2.
'Limit' and 'skip' are optional parameters
By default if you omit the limit parameter only the first 1000 docs are displayed
#>
Find-LiteDBDocument -Collection movies -Limit 5 -Skip 2
2️⃣▶️ Find by SQL Query
# get the first 5 documents from the movies collection
Find-LiteDBDocument movies -Sql "Select $ from movies limit 5"

# get the first 5 documents with selected properties or fields from the movies collection
Find-LiteDBDocument movies -Sql "Select _id,Title from movies limit 5"

# using where to filter the results - string filter
Find-LiteDBDocument movies -Sql "Select _id,Title from movies Where MPAA = 'PG-13'"

# using where to filter the results - greaterthan
Find-LiteDBDocument movies -Sql "Select _id,Title from movies Where Rating > 7.5"

# using multiple where filters. ( movies that contain 'talking' in their title)
Find-LiteDBDocument movies -Sql "Select _id,Title from movies where MPAA = 'PG-13' and Title like '%talking'"

# Sorting by name descending
Find-LiteDBDocument movies -Sql "Select _id,Title from movies where MPAA = 'PG-13' order by Title desc"

# date filter
Find-LiteDBDocument movies -Sql "select _id,Title,ReleaseDate from test where ReleaseDate > datetime('8/16/1989') order by Releasedate desc"

# using Functions
# get the first 5 documents with selected properties or fields from the movies collection
Find-LiteDBDocument movies -Sql "Select upper(Title),_id,MPAA from movies limit 5"
3️⃣▶️ Find by Named Queries
# Wildcard filter B*. Select 2 properties _id & status to display in the output
# Select is a mandatory parameter when used with -Where
Find-LiteDBDocument movies -Where "Title like '%talking'" -Select "{_id,Title,MPAA}"

# Calculated properties: show Title as MovieName
Find-LiteDBDocument movies -Where "Title like '%talking'" -Select "{_id,MovieName: Title}"

# limit: output to 2 documents
Find-LiteDBDocument movies -Where "Title like '%s%'" -Select "{_id,MovieName: Title}" -Limit 2

# Skip: skip first 2 documents
Find-LiteDBDocument movies -Where "Title like '%s%'" -Select "{_id,MovieName: Title}" -Limit 2 -Skip 2

# using Functions
Find-LiteDBDocument movies -Where "Title like '%talking'" -Select "{Name: _id,MovieName : UPPER(Title)}" -Limit 2

# for a list of other functions refer to : http://www.litedb.org/docs/expressions/
4️⃣▶️ Find All Documents

By default when used with no other parameters the cmdlet lists all documents in the collection.

Find-LiteDBDocument movies

🪲 Update records

1️⃣▶️ Update by SQL Query
#update multiple fields with a wildcard where query
Update-LiteDBDocument 'movies' -sql "UPDATE movies SET Title='Turner and Hooch',Runtime = 101 where _id=3"

lets stop the BITS service and then update the collection with the new status

2️⃣▶️ Update by Id
$mymovie = [PSCustomObject]@{
    Title = 'Turner and Hooch'
    _id = 3
    Budget = [int64]13000001
    Gross = [int64]71079915
    MPAA = 'PG'
    Rating = 7.2
    RatingCount = 91415
    ReleaseDate = [Datetime]'7/28/1989'
    RunTime = 101
    }

    $mymovie |  ConvertTo-LiteDbBSON | Update-LiteDBDocument -Collection movies

# retrieve the movie record in the litedb collection to see the updated status
Find-LiteDBDocument -Collection movies -ID 3

Output:


Collection  : movies
_id         : 3
Title       : Turner and Hooch
ReleaseDate : 7/28/1989 12:00:00 AM
Budget      : 13000001
Rating      : 7.2
RatingCount : 91415
Gross       : 71079915
MPAA        : PG
RunTime     : 101
3️⃣▶️ Update by BsonExpression

The where statement is a predicate or condition which will determine the documents to be updated.

# set the status property of the service named bfe to 4
Update-LiteDBDocument movies -Set "{Runtime:101}"  -Where "_id = 3"

# Retrieve all documents whose displayname begins with blue and Transform the name property to uppercase
Update-LiteDBDocument movies -set "{Title:UPPER(Title)}" -Where "Title like '%talking'"

⛔ Delete Records

1️⃣▶️ Delete by Id
# Delete record by ID
Remove-LiteDBDocument -Collection movies -ID 3

# If we try to retrieve the record we should see a warning
 Find-LiteDBDocument -Collection movies -ID 3
WARNING: Document with ID ['3'] does not exist in the collection ['movies']
2️⃣▶️ Delete by BsonExpression
# delete all records from the movies collection where the property Budget is null
Remove-LiteDBDocument movies -Query "Budget = null"
3️⃣▶️ Delete by SQL Query
#Deleteall records from the service collectionwhose displayname matches'xbox live'
Remove-LiteDBDocument 'movies' -Sql "delete movies where Title like '%talking'"

🌅 Upsert Records

Upsert stands for - Add if missing else update existing.

$movies = $dataset |
    ConvertFrom-Csv |
    Select-Object @{Name = "_id"; E = { [int]$_.MovieID } },
    @{Name = "Budget"; E = { [int64]$_.Budget } },
    @{Name = "Gross"; E = { [int64]$_.Gross } },
    @{Name = "ReleaseDate"; E = { [datetime]$_.'Release Date' } },
    @{Name = "RunTime"; E = { [int]$_.Runtime } },
    @{Name = "Rating"; E = { [Double]$_.Rating } },
    @{Name = "RatingCount"; E = { [int64]$_.RatingCount } },
    Title, MPAA -First 15

$movies |
    ConvertTo-LiteDbBSON |
    Upsert-LiteDBDocument -Collection movies

Query Filters

Using Query filters is not recomended anymore, it may be deprecated in future.


🚕 Close LiteDB Connection

Close-LiteDBConnection

WIKI

pslitedb's People

Contributors

tanyasweet1933 avatar v2kiran avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

pslitedb's Issues

"Schema" not getting updated

Having some issues with the "schema" not being updated.
I have inserted some documents, then I enrich them (find them, update a local copy - sometimes with extra fields, delete the old document, insert the updated document with new field).
The problem is, the GUI tools and select $ all returns only the initial fields of the documents inserted.
We need a way to either update the "Schema" - Or inserts should check if new fields are present.

If I do a select where NewField > '', then my results includes the NewField, since it is present in first (?) or all records. If I do a select $ on all records, I do NOT get the new fields in the response.

Of course selecting individual documents by uniq search criteria will return all fields, since the one document have them.

Getting an error Process is used by another process

I am using PSLiteDB in my Project, it is awesoem and helpful for my Project. But there is one bug. As mentioned that LiteDB is locked unless we Close the connection. Sometimes, even when i try to open the Open-LiteDBConnection still showing that Process is used by another process. And Even try to close using Close-LiteDBConnection but it shows that no Open-LiteDBConnection Found.

Please guide me where i'm doing wroung.

Thank you

String is always converted to DateTime under certain conditions

Hi,
I have a problem with the latest version of PSLiteDB. When I want to convert a PSCustomObject to a LiteDBBson Object, I always get the following error:
"Clould not convert string to DateTime: Datenschutz"

As I already found out, the problem is the string itself. As soon as the string contains the word DATE, it is recognized as a date. In German data protection means DATEnschutz and after here Date is in the word, it tries to convert the string to a date.

small example code:

Import-Module PSLiteDB

$dbPath = "C:\tmp\test1.db"
New-LiteDBDatabase -Path $dbPath -Verbose
Open-LiteDBConnection -Database $dbPath
New-LiteDBCollection -Collection Event

[String]$mystring = 'Datenschutz'

$myOBJ = [PSCustomObject]@{
 MyString = $mystring
}

$myBSON = $myOBJ | ConvertTo-LiteDbBson

ConvertTo-LiteDbBSON failed to serialise '_id' in ObjectID type

Description

I have an issue with $array | ConvertTo-LiteDbBSON -As Array, where the array items has _id in ObjectID type, ObjectID type is the default index type when the LiteDB is created by New-LiteDBCollection

Reference: LiteDB / Data Structure / Objectid (https://www.litedb.org/docs/data-structure/#objectid)

Expected Behavior

# 1. Connect to LiteDB, find some document, save to an variable
Open-LiteDbConnection -Database $DBPath -Credential $DBCred
$array = Find-LiteDBDocument -Collection $DBCollection -Limit 2

# Example values
array[0]
_id: 63802bc8ac6e08130fe7df92
name: Apple

# 2. Update an item in the array
$array[0].name = 'Banana'

# 3. Push update to DB
$array_BSON = $array | ConvertTo-LiteDbBSON -As Array
Update-LiteDBDocument -Collection $DBCollection -BsonDocumentArray $array_BSON

Actual Behavior

$array_BSON = $array | ConvertTo-LiteDbBSON -As Array
WARNING: Unable to convert value [[_id, {"Timestamp":1669344201,"Machine":11300360,"Pid":4879,"Increment":15227919,"CreationTime":"2022-11-25T02:43:21Z"}]] to datetime
  1. Update-LiteDBDocument does not throw warnings of _id not found
  2. DB not updated

Specifications

  • Version: PSLiteDB committed on 2022 Feb 13 (latest as of writing)
  • Platform: Powershell 7.3.0
  • Subsystem:

Provide C# source for PSLiteDB.dll

Hi Kiran,

Any plan/interest in opening up your C# code for PSLiteDB.dll (i.e the whole solution's source code)?

I'm interested to use the module but I am concerned about the lack of sourc ode for the DLL.

I'd be happy to share any fixes/features in the future if any.

Thanks!

License?

Under which license is PSLiteDB available?

dbref feature

hi @v2kiran

Just wondering whether there is dbref feature out of box I can use to include or link other collection?

Thanks
Eric Xin

Converting to string error. PSLiteDB 2.1.0

Hi Team,

I am using PSLite 2.1.0. Updated litedb to the latest available version 5.0.9
Trying to import string value which contains the world "Long" failed.

Collection name is PublicTeamsTeamCln

$teamRecord = @{GroupId=324186ae-00a0-41be-a50d-9292830900a1; GroupDisplayName= "MyGroup"; Alias=""; GroupOwner="Kai-Long Mister"; GroupOwnerUPN="[email protected]"; Members=6; CreatedDateTime=09/18/2019 13:16:51}

$o = $teamRecord | Select-Object @{ Name = "_id"; E = { [System.String]$teamRecord.GroupId} },
@{ Name = "GroupDisplayName"; E = { [System.String]$teamRecord.GroupDisplayName } },
@{ Name = "Alias"; E = { [System.String]$teamRecord.Alias } },
@{ Name = "GroupOwner"; E = { [System.String]$teamRecord.GroupOwner } },
@{ Name = "GroupOwnerUPN"; E = { [System.String]$teamRecord.GroupOwnerUPN } },
@{ Name = "Members"; E = { [int]$teamRecord.Members } },
@{ Name = "CreatedDateTime"; E = { [datetime]$teamRecord.CreatedDateTime } }

$o | ConvertTo-LiteDbBSON | Upsert-LiteDBDocument -Collection PublicTeamsTeamCln

The error
Failed to find document in Db: Impossible to convert value « Kai-Long Mister » in type « System.Int64 ». Erreur : « incorrect input string format »

Please, give a help,
Regards

Update-LiteDBDocument is failing

Using the example presented in the README and other testing code (https://cadayton.keybase.pub/PSGallery/Scripts/psConsole.html), the Update-LiteDBDocument command is consistently failing with the same error message.

PS>Get-Service BITS |
>>   Select @{Name="_id";E={$_.Name}},DisplayName,Status,StartType |
>>         ConvertTo-LiteDbBSON |
>>         Update-LiteDBDocument -Collection SvcCollection
Update-LiteDBDocument:
Line |
   4 |          Update-LiteDBDocument -Collection SvcCollection
     |          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | The input object cannot be bound to any parameters for the command either because the command does not take pipeline input or the input and its properties do not match any of the parameters that take pipeline input.

Open-LiteDBConnection memory db

Hi @v2kiran

If in the beginning, the open litedbconnection is pointing to the in-memory db, and whether I still have chance to save the result to file finally if I change my original mind?

Thanks
Eric Xin

February 28th

Hi Kiran,

Incredible.
An object with datetime 28/02/2020 11:12:15, cannot be convert into BsonDate, line 3, position 28.
May be this come from LiteDB.

Error
Could not convert string to DateTime. Path BsonDate, line 3, position 28.

Any idea !

Import-Module fails

Description

When I'm trying to import pslitedb module it fails with error

Import-Module : Could not load file or assembly 'LiteDB, Version=5.0.9.0, Culture=neutral, PublicKeyToken=4ee40123013c9
f27' or one of its dependencies. The system cannot find the file specified.
At line:1 char:1
+ Import-Module pslitedb
+ ~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Import-Module], FileNotFoundException
    + FullyQualifiedErrorId : System.IO.FileNotFoundException,Microsoft.PowerShell.Commands.ImportModuleCommand

Steps to reproduce

  1. Install-Module PSLiteDB or Install-Module PSLiteDB -Scope AllUsers
  2. Import-Module PSLiteDB
  3. Error.

System info

Windows 10 or Windows Server 2019
Powershell 5
PSLiteDB 2.2.0

insert file

Hi @v2kiran

Thanks for doing a good job for this great module.
Just wondering whether I can add/upload file as well?

Thanks
Eric Xin

ConvertTo-LiteDbBson - how to use long, datatime

I use LiteDB in C# and I am able to create classes with Long (Int64) and DateTime date types and insert them into my db collection.

Now I need to do the same in PowerShell so I am trying PSLiteDB.

However I get errors on ConvertTo-LiteDbBson when I do.

Here's an example. Works unless I use the long or int64 and datetime type values.

I'm looking for guidance in getting this example to accept a long (specifically a datetime.tick) and a datetime).

Pls advise.


function Main()
{

Import-Module C:\PSScripts\PSLiteDB-master\PSLiteDB

$dbPath = "C:\temp\test1.db"
New-LiteDBDatabase -Path $dbPath -Verbose
Open-LiteDBConnection -Database $dbPath
New-LiteDBCollection -Collection Event

[Int]$myint = 1
[Long]$mylong = (Get-Date).Ticks
[String]$mystring = "foo"
[DateTime]$mydate = (Get-Date)
[Bool]$mybool = $true

$myOBJ = [PSCustomObject]@{
    MyInt = $myint
    MyLong = $mylong
    MyString = $mystring
    MyDate = $mydate
    MyVool = $mybool
}


$myBSON = $myOBJ | ConvertTo-LiteDbBson

Add-LiteDBDocument -Collection Event -Document $myBSON

Find-LiteDBDocument -Collection Event

}

Main


These are the errors I get when I do.


Exception calling "Deserialize" with "1" argument(s): "Value was either too large or too small for an Int32."
At C:\PSScripts\PSLiteDB-master\PSLiteDB.psm1:37 char:17

  •             $obj = [LiteDB.JsonSerializer]::Deserialize(
    
  •             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    
    • CategoryInfo : NotSpecified: (:) [], MethodInvocationException
    • FullyQualifiedErrorId : OverflowException

Add-LiteDBDocument : Cannot validate argument on parameter 'Document'. The argument is null or empty. Provide an argument that is not null or empty, and then try the command again.
At C:\PSScripts\EnableLiteDBTest.ps1:28 char:52

  • Add-LiteDBDocument -Collection Event -Document $myBSON
    
  •                                                ~~~~~~~
    
    • CategoryInfo : InvalidData: (:) [Add-LiteDBDocument], ParameterBindingValidationException
    • FullyQualifiedErrorId : ParameterArgumentValidationError,PSLiteDB.AddLiteDBDocument

Unable to find type [PSLiteDB.Helpers.MSJsonDateConverter1].
At C:\PSScripts\PSLiteDB-master\PSLiteDB.psm1:51 char:47

  • ... $hash[$kvp.key] = [PSLiteDB.Helpers.MSJsonDateConverter1]::Conver ...
  •                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    
    • CategoryInfo : InvalidOperation: (PSLiteDB.Helpers.MSJsonDateConverter1:TypeName) [], RuntimeException
    • FullyQualifiedErrorId : TypeNotFound

Array of complex objects is lost on save

I'm having the issue that when an array of say either hashtables or pscustomobjects is saved to a document, upon get, there is only one item :(.

example

$obj = [pscustomobject]@{
    prop = @(@{obb=123}),@{test=123})
    _id="1"
} | ConvertTo-LiteDbBSON

Add-LiteDBDocument -Collection test - Document $obj

Find-LiteDBDocument -Collection test

Collection        _id prop
----------        --- ----
test 2   @{Collection=test; test=123}               # pscustomobject example
test 1   {1,2,3}                                                    # hashtable example

Any help here would be appreciated.

thanks

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.