Windows PowerShell: How to parse the log file?

Question

I have an input file with below contents:

27/08/2020  02:47:37.365 (-0516)  hostname12    ult_licesrv       ULT  5  LiceSrv Main[108                    00000  Session 'session1' (from 'vmpms1\[email protected]') request for 1 additional licenses for module 'SA-XT' - 1 licenses have been allocated by concurrent usage category 'Unlimited' (session module usage now 1, session category usage now 1, total module concurrent usage now 1, total category usage now 1)
27/08/2020  02:47:37.600 (-0516)  hostname13    ult_licesrv       ULT  5  LiceSrv Main[108                    00000  Session 'sssion2' (from 'vmpms2\[email protected]') request for 1 additional licenses for module 'SA-XT-Read' - 1 licenses have been allocated by concurrent usage category 'Floating' (session module usage now 2, session category usage now 2, total module concurrent usage now 1, total category usage now 1)
27/08/2020  02:47:37.115 (-0516)  hostname141    ult_licesrv       CMN  5  Logging Housekee                    00000  Deleting old log file 'C:\Program Files\PMCOM Global\License Server\diag_ult_licesrv_20200824_011130.log.gz' as it exceeds the purge threashold of 72 hours
27/08/2020  02:47:37.115 (-0516)  hostname141    ult_licesrv       CMN  5  Logging Housekee                    00000  Deleting old log file 'C:\Program Files\PMCOM Global\License Server\diag_ult_licesrv_20200824_021310.log.gz' as it exceeds the purge threashold of 72 hours
27/08/2020  02:47:37.625 (-0516)  hostname150    ult_licesrv       ULT  5  LiceSrv Main[108                    00000  Session 'session1' (from 'vmpms1\[email protected]') request for 1 additional licenses for module 'SA-XT' - 1 licenses have been allocated by concurrent usage category 'Unlimited' (session module usage now 2, session category usage now 1, total module concurrent usage now 2, total category usage now 1)

I need to generate and output file like below:

Date,time,hostname,session_module_usage,session_category_usage,module_concurrent_usage,total_category_usage
27/08/2020,02:47:37.365 (-0516),hostname12,1,1,1,1
27/08/2020,02:47:37.600 (-0516),hostname13,2,2,1,1
27/08/2020,02:47:37.115 (-0516),hostname141,0,0,0,0
27/08/2020,02:47:37.115 (-0516),hostname141,0,0,0,0
27/08/2020,02:47:37.625 (-0516),hostname150,2,1,2,1

The output data order is: Date,time,hostname,session_module_usage,session_category_usage,module_concurrent_usage,total_category_usage.

Put 0,0,0,0 if no entry for session_module_usage,session_category_usage,module_concurrent_usage,total_category_usage

I need to get content from the input file and write the output to another file.

Update

I have created a file input.txt in F drive and pasted the log details into it. Then I form an array by splitting the file content when a new line occurs like below.

$myList = (Get-Content -Path F:\input.txt) -split '\n'

Now I got 5 items in my array myList. Then I replace the multiple blank spaces with a single blank space and formed a new array by splitting each element by blank space. Then I print the 0 to 3 array elements. Now I need to add the end values (session_module_usage,session_category_usage,module_concurrent_usage,total_category_usage).

PS C:\Users\user> $myList = (Get-Content -Path F:\input.txt) -split '\n'
PS C:\Users\user> $myList.Length
5
    PS C:\Users\user> $myList = (Get-Content -Path F:\input.txt) -split '\n'
PS C:\Users\user> $myList.Length
5
PS C:\Users\user> for ($i = 0; $i -le ($myList.length - 1); $i += 1) {
>> $newList = ($myList[$i] -replace '\s+', ' ') -split ' '
>> $newList[0]+','+$newList[1]+' '+$newList[2]+','+$newList[3]
>>  }
27/08/2020,02:47:37.365 (-0516),hostname12
27/08/2020,02:47:37.600 (-0516),hostname13
27/08/2020,02:47:37.115 (-0516),hostname141
27/08/2020,02:47:37.115 (-0516),hostname141
27/08/2020,02:47:37.625 (-0516),hostname150

@RanadipDutta I have updated the question with my latest findings, could you please have a look? — Sreejith Sree
– Sreejith Sree, Commented Sep 4, 2020 at 7:08
Removed the comment because the question has been updated. I will look into it in my free time by creating a sample file and the data. @Sreejith — Ranadip Dutta
– Ranadip Dutta, Commented Sep 4, 2020 at 7:17

Sreejith Sree · Accepted Answer · 2020-09-04 08:36:57Z

2

If you really need to filter on the granularity that you're looking for, then you may need to use regex to filter the lines.

This would assume that the rows have similarly labeled lines before the values you're looking for, so keep that in mind.

[System.Collections.ArrayList]$filteredRows = @()
$log = Get-Content -Path C:\logfile.log
foreach ($row in $log) {
    $rowIndex = $log.IndexOf($row)
    $date = ([regex]::Match($log[$rowIndex],'^\d+\/\d+\/\d+')).value
    $time = ([regex]::Match($log[$rowIndex],'\d+:\d+:\d+\.\d+\s\(\S+\)')).value
    $hostname = ([regex]::Match($log[$rowIndex],'(?<=\d\d\d\d\)  )\w+')).value
    $sessionModuleUsage = ([regex]::Match($log[$rowIndex],'(?<=session module usage now )\d')).value
    if (!$sessionModuleUsage) {
        $sessionModuleUsage = 0
    }
    $sessionCategoryUsage = ([regex]::Match($log[$rowIndex],'(?<=session category usage now )\d')).value
    if (!$sessionCategoryUsage) {
        $sessionCategoryUsage = 0
    }
    $moduleConcurrentUsage = ([regex]::Match($log[$rowIndex],'(?<=total module concurrent usage now )\d')).value
    if (!$moduleConcurrentUsage) {
        $moduleConcurrentUsage = 0
    }
    $totalCategoryUsage = ([regex]::Match($log[$rowIndex],'(?<=total category usage now )\d')).value
    if (!$totalCategoryUsage) {
        $totalCategoryUsage = 0
    }
    $hash = [ordered]@{
        Date = $date
        time = $time
        hostname = $hostname
        session_module_usage = $sessionModuleUsage
        session_category_usage = $sessionCategoryUsage
        module_concurrent_usage = $moduleConcurrentUsage
        total_category_usage = $totalCategoryUsage
    }
    $rowData = New-Object -TypeName 'psobject' -Property $hash
    $filteredRows.Add($rowData) > $null
}
$csv = $filteredRows | convertto-csv -NoTypeInformation -Delimiter "," | foreach {$_ -replace '"',''}
$csv | Out-File C:\results.csv

What essentially needs to happen is that we need to get-content of the log, which returns an array with each item terminated on a newline.

Once we have the rows, we need to grab the values via regex Since you want zeroes in some of the items if those values don't exist, I have if statements that assign '0' if the regex returns nothing

Finally, we add each filtered item to a PSObject and append that object to an array of objects in each iteration.

Then export to a CSV.

edited Sep 4, 2020 at 8:36

Sreejith Sree

3,9259 gold badges43 silver badges119 bronze badges

answered Sep 4, 2020 at 7:17

Andrew Ryan Davis

6591 gold badge5 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Joey Over a year ago

If the repeated Add-Member calls here are too verbose, I suggest using an Add-Member -PassThru chain via the pipeline instead. Passing -InputObject via arguments is often unnecessary for things can be easily be piped.

Sreejith Sree Over a year ago

I tried this and working as expected, but could you please remove all the double-quotes and can you make the command a single line.

Andrew Ryan Davis Over a year ago

Modified to reflect what you're looking for

Sreejith Sree Over a year ago

It's working thanks but got an error for foreach, so I have edited the answer with the fix

Joey · Accepted Answer · 2020-09-04 08:33:07Z

You can probably pick apart the lines with a regex and substrings easily enough. Basically something like the following:

# Iterate over the lines of the input file
Get-Content F:\input.txt |
    ForEach-Object {
      # Extract the individual fields
      $Date = $_.Substring(0, 10)
      $Time = $_.Substring(12, $_.IndexOf(')') - 11)
      $Hostname = $_.Substring(34, $_.IndexOf(' ', 34) - 34)
      $session_module_usage = 0
      $session_category_usage  = 0
      $module_concurrent_usage = 0
      $total_category_usage = 0
      if ($_ -match 'session module usage now (\d+), session category usage now (\d+), total module concurrent usage now (\d+), total category usage now (\d+)') {
          $session_module_usage = $Matches[1]
          $session_category_usage  = $Matches[2]
          $module_concurrent_usage = $Matches[3]
          $total_category_usage = $Matches[4]
      }
      # Create custom object with those properties
      New-Object PSObject -Property @{
          Date = $Date
          time = $Time
          hostname = $Hostname
          session_module_usage = $session_module_usage
          session_category_usage = $session_category_usage
          module_concurrent_usage = $module_concurrent_usage
          total_category_usage = $total_category_usage
      }
    } |
    # Ensure column order in output
    Select-Object Date,time,hostname,session_module_usage,session_category_usage,module_concurrent_usage,total_category_usage |
    # Write as CSV - without quotes
    ConvertTo-Csv -NoTypeInformation |
    ForEach-Object { $_ -replace '"' } |
    Out-File F:\output.csv

Whether to pull the date, time, and host name from the line with substrings or regex is probably a matter of taste. Same goes for how strict the format must be matched, but that to me mostly depends on how rigid the format is. For more free-form things where different lines would match different regexes, or multiple lines makes up a single record, I also quite like switch -Regex to iterate over the lines.

You are allowed to take a look at the code, understand it, and figure out the problem yourself, you know.

Collectives™ on Stack Overflow

Windows PowerShell: How to parse the log file?

2 Answers 2

4 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related