[PackageTool.ps1] Conversion from JSON failed with error: Unterminated string. about storebroker HOT 16 CLOSED

I made the suggested changes and do not generate a PDP, but the failure still occurs. See https://github.com/LanceMcCarthy/MediaFileManager/runs/980649893?check_suite_focus=true

That's what I would have expected. That suggestion wasn't to fix the problem, I was simply trying to simplify your script; as you indicated below however, that goes against the goal you were trying to achieve.

I am also trying to understand exactly how the API works and what it expects. I'm about 80% done with developing a TypeScript version of the Store Broker that is on npm. Here are the tasks https://github.com/LanceMcCarthy/PartnerCenterBroker/blob/main/src/index.ts

There is an Azure DevOps task written in TypeScript that you could look at for reference (or use).

Something for you to consider: As you might have noticed with this project, there has been minimal activity on the master (v1) branch. The same can't be said of StoreBroker v2. StoreBroker v2 is built on top of the next version of the Windows Submission API (v2) which is substantially different than v1, and intended to eventually replace it. Unfortunately, v2 isn't publicly available yet, and we still don't know what that timeframe would be for its public release. StoreBroker v2 has been in-use internally now for over 1.5 years, and there's a new version of the Azure DevOps task that is built on top of StoreBroker v2, but that too is only available internally until such time as the new API is made widely available publicly.

from storebroker.

Thanks @LanceMcCarthy. I think the easiest way to debug what might be going on is to understand your submission that it's trying to convert into a config file. Can you run the following?

$appId = '9PD3JFK7W5MB'
$app = Get-Application -AppId $appId
$sub = Get-ApplicationSubmission -AppId $appId -SubmissionId $($app.lastPublishedApplicationSubmission.id)
$subJsonFilePath = New-TemporaryFile
ConvertTo-Json -InputObject $sub -Depth 20 | Out-File -FilePath $subJsonFilePath
Write-Information -MessageData "Please upload $($subJsonFilePath.FullName) to a new gist at https://gist.github.com/LanceMcCarthy and then reference that gist in microsoft/StoreBroker#197" -InformationAction Continue

That would allow us to reproduce the behavior of Get-StoreBrokerConfigFileContentForAppId without needing access to your developer account. Something weird is going on there, as that command should take ~5 seconds to complete.

As an aside, the general usage pattern for StoreBroker would be to generate the config file once and check that in with your source code. Same with the PDP file content. The intention of the config file is two fold:

1. Let you define common parameters to New-SubmissionPackage so that they don't need to be specified at the command-line
2. Allow you to alter app-wide settings that have no command-line component (like changing the pricing, category, etc...).

If you're always generating it fresh, you're not getting any of those benefits. That's fine if that functionality isn't desired, but wanted to make sure that was an explicit decision on your part.

And as for the PDP content, like I said before, you'd convert an existing published submission into the individual components (XML and image files) once and then store them with your code, and then update the listing metadata in those files locally when you've made changes to the app that you're intending to publish. Then you'd generate a submission package with the updated PDP content and make use of -UpdateListings when calling Update-AppSubmission (with or without -AddPackages/-ReplacePackages).

You may just want to add a comment to that effect (that those command are intended to only be run once when initially configuring StoreBroker).

from storebroker.

Is this planned to merge soon? I noticed you have 2 PRs open.

Done. One PR was for master (StoreBroker v1), and the other was for the v2 branch (which targets the v2 API).

from storebroker.

Thanks for the report, @LanceMcCarthy.

Would you be willing to share out the config file that was generated at $brokerConfigFilePath? The issue is with the notesForCertification that was injected into there, and so I'd like to understand what's going on there.

As an aside, you can completely skip the "Generating PDP" step. Just don't include the PDPRootPath parameter for New-SubmissionPackage. Since you're just doing package updates, there's no need to include any of the metadata. The metadata is only useful when you're actually trying to update it.

from storebroker.

Note for maintainers...relevant code for inspection:

from storebroker.

It's not clear to me why this isn't calling ConvertTo-Json on the result of Get-EscapedJsonValue (or instead of calling that entirely).

from storebroker.

Thanks @HowardWolosky

I made the suggested changes and do not generate a PDP, but the failure still occurs. See https://github.com/LanceMcCarthy/MediaFileManager/runs/980649893?check_suite_focus=true

In that same run, I did attempt to output the SBconfig.json contents so I could copy/paste it, but I'm not sure it is even reaching that step because I'm not seeing the output. I will try to upload the file to a blob instead.

Metadata

You're right that I dont have any changes to the metadata in this current iteration, but I plan on building it in because that repo is intended as a "gold standard" for dev community to reference for when they want to see a real-world GitHub Action for deploying a WPF/UWP app to both side-load distributions.

The step prior to the StoreBroker is my own custom GH Action that uploads the .appinstaller (and associated packages) to Azure Storage blob. The Store Broker step is the next phase that distributes to the Store (I'll add a flights branch when this works).

Side Note - Building a new Broker

I am also trying to understand exactly how the API works and what it expects. I'm about 80% done with developing a TypeScript version of the Store Broker that is on npm. Here are the tasks https://github.com/LanceMcCarthy/PartnerCenterBroker/blob/main/src/index.ts

I have studied the API docs and understand how complicated the payloads can get (the number of nested objects gets gnarly at times https://github.com/LanceMcCarthy/PartnerCenterBroker/blob/main/src/interfaces.ts) so I like the idea of cloning a previous submission and have done that here https://github.com/LanceMcCarthy/PartnerCenterBroker/blob/36901851ace5126d562ed619a6f3c2446831aa85/src/helpers.ts#L25-L47 .

I aim to provide developers with options, no matter what their environment is (at least until PS Core works 100%)

from storebroker.

That's what I would have expected. That suggestion wasn't to fix the problem, I was simply trying to simplify your script; as you indicated below however, that goes against the goal you were trying to achieve.

Understood, I was just sharing the end result. Thank you for the suggestion as it saves another possible point of failure.

There is an Azure DevOps task written in TypeScript that you could look at for reference (or use).

Thank you! Although it seems I should not put too much effort into using the v1 endpoints, but if v2 endpoints aren't public yet my options are limited and might as well keep moving forward with it. I can then do a v2 of my package later.

I reviewed the code and I do already have everything they implemented, and more :) What it did show me is that I can simplify the metadata operations and just do the clone+add package and forget about giving the user granular capability to change metadata.

Something for you to consider: As you might have noticed with this project, there has been minimal activity on the master (v1) branch. The same can't be said of StoreBroker v2. StoreBroker v2 is built on top of the next version of the Windows Submission API (v2) which is substantially different than v1, and intended to eventually replace it. Unfortunately, v2 isn't publicly available yet, and we still don't know what that timeframe would be for its public release. StoreBroker v2 has been in-use internally now for over 1.5 years, and there's a new version of the Azure DevOps task that is built on top of StoreBroker v2, but that too is only available internally until such time as the new API is made widely available publicly.

In my initial research, I did find the Partner Center APIs which heavily use the MS Graph, but do not see any endpoints for App submission :( This is why my current efforts are still using manage.devcenter.microsoft.com/v1/ instead

Is it going to be manage.devcenter.microsoft.com/v2/ or is it going to use https://api.partnercenter.microsoft.com endpoints?

NDA Help

Although it doesn't sound like you need the testing for v2 of the submission API, but if you do, we can use my MVP NDA for my private Actions 😉 (I would help write day1 blog (similar to my Windows 10 on ARM: Day-One Developer Experience post). I am also under several other TAP programs and have pre-approved clearance for future TAPs/NDAs.

This would not only help me, but I can also have working examples for the community ready-to-go.

from storebroker.

@HowardWolosky Okay, I tried making this as simple and straightforward as possible by using all full strings where possible and broke out the steps to different stages to catch where it is failing

The Store Broker now seems to hang when trying to generate the SBConfig file (for 26 minutes until the VM quits)

************************* GETS STUCK HERE *******************************
# ---- Create config file  
New-StoreBrokerConfigFile -AppId $env:PartnerCenterStoreId -Path 'D:\a\MediaFileManager\MediaFileManager\SBTemp\SBConfig.json'

Here are the relevant steps, you can see the offender in Part 2

 # ************ PART 1  ************ # 
- name: Install Store Broker (SB)
  uses: Amadevus/pwsh-script@v1
  id: install-store-broker
  with:
    script: |
      # ---- Install StoreBroker
      Set-ExecutionPolicy RemoteSigned -Force
      git clone https://github.com/Microsoft/StoreBroker.git 'D:\a\MediaFileManager\MediaFileManager\SBGitRoot\'

# ************ PART 2 ************ #
- name: Authenticate SB and Generate SBConfig.json
  uses: Amadevus/pwsh-script@v1
  id: authenticate-store-brokwer
  with:
    script: |
      $username = $env:PartnerCenterClientId
      $password = ConvertTo-SecureString $env:PartnerCenterClientSecret -AsPlainText -Force
      $cred = New-Object System.Management.Automation.PSCredential ($username, $password)
      Import-Module -Force 'D:\a\MediaFileManager\MediaFileManager\SBGitRoot\StoreBroker'

      # ---- Authenticating with Service Principal Credentials
      Set-StoreBrokerAuthentication -TenantId $env:PartnerCenterTenantId -Credential $cred
      
      # ---- Making a new directory for Store Broker generated files
      New-Item -Path 'D:\a\MediaFileManager\MediaFileManager\SBTemp' -ItemType Directory

******** GETS STUCK HERE (for 26 minutes) ****************
      # ---- Create config file  
      New-StoreBrokerConfigFile -AppId $env:PartnerCenterStoreId -Path 'D:\a\MediaFileManager\MediaFileManager\SBTemp\SBConfig.json'


# ************ PART 3 (debug step to upload the sbconfig.json file to my blob) ************
- name: Uploading SBconfig.json to Azure Blob (temp)
  id: blob-upload-sbconfig
  uses: LanceMcCarthy/[email protected]
  with:
    connection_string: ${{ secrets.AZURE_DVLUP_BLOB_CONNECTION_STRING }}
    container_name: general-app-files
    source_folder: D:\a\MediaFileManager\MediaFileManager\SBTemp\
    destination_folder: SBTemp

# ************ PART 4 do the actual submission ************
- name: Store Broker (Upload the Store build to Microsoft Partner Center)
  uses: Amadevus/pwsh-script@v1
  id: store-broker-submit-app
  with:
    script: |
       $uploadPackage = "D:\a\MediaFileManager\MediaFileManager\src\MediaFileManager\PackageProject\StoreUploadPackages\PackageProject_" + 
$env:UWP_VERSION + "_" + $env:UwpBundlePlatform + "_bundle.appxupload"
        $username = $env:PartnerCenterClientId
        $password = ConvertTo-SecureString $env:PartnerCenterClientSecret -AsPlainText -Force
        $cred = New-Object System.Management.Automation.PSCredential ($username, $password)
          
        Import-Module -Force 'D:\a\MediaFileManager\MediaFileManager\SBGitRoot\StoreBroker'
          
          # ---- Authenticating with Service Principal Credentials
        Set-StoreBrokerAuthentication -TenantId $env:PartnerCenterTenantId -Credential $cred
    
        # ---- Create submissionPackage
        New-SubmissionPackage -ConfigPath 'D:\a\MediaFileManager\MediaFileManager\SBTemp\submission.json' -ImagesRootPath 'D:\a\MediaFileManager\MediaFileManager\src\MediaFileManager\PackageProject\Images' -AppxPath $uploadPackage -OutPath 'D:\a\MediaFileManager\MediaFileManager\SBTemp\' -OutName 'submission.json'
      
        # ---- Submitting app, we only want to close last submission and just update package
        Update-ApplicationSubmission -ReplacePackages -AppId $env:PartnerCenterStoreId -SubmissionDataPath 'D:\a\MediaFileManager\MediaFileManager\SBTemp\submission.json' -PackagePath $uploadPackage -AutoCommit -Force

Output

Here is the out put from step 2 (with repeated lines removed)

- Elapsed: 0 second(s) [Getting access token]
**** REDACTED ****
DONE - Operation took 2 second(s) [Getting access token]

Elapsed: 0 second(s) [Getting data for AppId: 9PD3JFK7W5MB]
**** REDACTED ****
DONE - Operation took 3 second(s) [Getting data for AppId: 9PD3JFK7W5MB]
2020-08-13 22:19:15 : runneradmin : Telemetry is currently enabled.  It can be disabled by setting "$global:SBDisableTelemetry = $true". Refer to USAGE.md#telemetry for more information.


-  Elapsed: 0 second(s) [Retrieving nuget package: Microsoft.Bcl.Async]
**** REDACTED ****
DONE - Operation took 4 second(s) [Retrieving nuget package: Microsoft.Bcl.Async]
2020-08-13 22:19:21 : runneradmin : To avoid this download delay in the future, copy the following file:
  [C:\Users\RUNNER~1\AppData\Local\Temp\ecf83896-abd9-4c21-b5eb-ea9e170df8ff\Microsoft.Bcl.Async.1.0.168\lib\net40\Microsoft.Threading.Tasks.dll]
either to:
  [D:\a\MediaFileManager\MediaFileManager\SBGitRoot\StoreBroker]
or to:
  a directory of your choosing, and save that directory path to $SBAlternateAssemblyDir
  

|  Elapsed: 0 second(s) [Retrieving nuget package: Microsoft.Diagnostics.Tracing.EventSource.Redist]
**** REDACTED ****
DONE - Operation took 2 second(s) [Retrieving nuget package: Microsoft.Diagnostics.Tracing.EventSource.Redist]
2020-08-13 22:19:23 : runneradmin : To avoid this download delay in the future, copy the following file:
  [C:\Users\RUNNER~1\AppData\Local\Temp\ecf83896-abd9-4c21-b5eb-ea9e170df8ff\Microsoft.Diagnostics.Tracing.EventSource.Redist.1.1.24\lib\net35\Microsoft.Diagnostics.Tracing.EventSource.dll]
either to:
  [D:\a\MediaFileManager\MediaFileManager\SBGitRoot\StoreBroker]
or to:
  a directory of your choosing, and save that directory path to $SBAlternateAssemblyDir
  

|  Elapsed: 2 second(s) [Retrieving nuget package: Microsoft.ApplicationInsights]
**** REDACTED ****
DONE - Operation took 2 second(s) [Retrieving nuget package: Microsoft.ApplicationInsights]
2020-08-13 22:19:26 : runneradmin : To avoid this download delay in the future, copy the following file:
  [C:\Users\RUNNER~1\AppData\Local\Temp\ecf83896-abd9-4c21-b5eb-ea9e170df8ff\Microsoft.ApplicationInsights.2.0.1\lib\net45\Microsoft.ApplicationInsights.dll]
either to:
  [D:\a\MediaFileManager\MediaFileManager\SBGitRoot\StoreBroker]
or to:
  a directory of your choosing, and save that directory path to $SBAlternateAssemblyDir
  
 -  Elapsed: 0 second(s) [Getting data for AppId: 9PD3JFK7W5MB SubmissionId: 1152921505688863279]
**** REDACTED ****
DONE - Operation took 9 second(s) [Getting data for AppId: 9PD3JFK7W5MB SubmissionId: 1152921505688863279]

Here's the run if you want to see the whole thing https://github.com/LanceMcCarthy/MediaFileManager/runs/982539639?check_suite_focus=true

from storebroker.

It definitely a problem inside the submission data being downloaded from DevCenter.

See the latest runs output, even after trying to copy out the data https://github.com/LanceMcCarthy/MediaFileManager/actions/runs/208040011

I'm going to try this on my normal pc and get real files I can include with source control. I'll let you know how that goes tomorrow

from storebroker.

No, that error indicates a poorly formatted config file. That means it already successfully downloaded the data and successfully created a config file. Now it's trying to read and covert the config file back into an object during New-SubmissionPackage, and it's failing at that conversion step:

Like I said, if you just run the commands in my previous response, I'll be able to see your submission and understand what's going on when the config file is getting created and why it's malformed.

from storebroker.

@HowardWolosky I just couldn't get GitHub actions to cooperate and give me the files, so I did everything locally on my dev machine instead.

You can get the generated file here https://dvlup.blob.core.windows.net/ci-cd/broker-output/tmp828.tmp

from storebroker.

@HowardWolosky Discovered the issue!

I think I found the culprit. In the notes for certification, I see a < character that is encoded in one place, but not the other.

SubmissionData.json

Notice the \u003c character in the release notes.

SBConfig.json

If you look at the sbconfig.json, the < character is not encoded.

Source Control-based Config

As you recommended, I checked in the config.json and submission.json files into my source. I also moved my Store Broker script into its own ps1 file. https://github.com/LanceMcCarthy/MediaFileManager/tree/main/.scripts.

My question now is: If I use the checked-in SubmissionData.json file, how does Update-ApplicationSubmission know to use the new APPX? Does it automatically change the packages list when I use -ReplacePackages flag?

For example:

# Generated (STILL FAILS b/c of BAD CHAR)
$newSubmissionDataPath = Join-Path -Path $sbTempFolderPath -ChildPath 'submissionData.json'

# Checked in with source control and has manually fixed notes
$originalSubmissionDataFilePath = 'D:\a\MediaFileManager\MediaFileManager\.scripts\submissionData.json'

$packagePath = 'D:\a\MediaFileManager\MediaFileManager\src\MediaFileManager\PackageProject\StoreUploadPackages\StoreUploadPackages.zip'

Update-ApplicationSubmission -ReplacePackages -AppId $env:PartnerCenterStoreId -SubmissionDataPath $originalSubmissionDataFilePath -PackagePath $packagePath -AutoCommit -Force

I did try using $originalSubmissionDataFilePath for the submission, but it failed as well with the same error. I'm not sure what went wrong there.

Next Steps

After multiple attempts to get past this, I think I'm going to submit a new version via DevCenter and clean out the cert notes After that is published, I can generate a new submissionData file for future use.

PS - Despite these issues in GitHub Actions (PS core?) StoreBroker is an absolute pleasure in user-mode locally.

from storebroker.

Thanks Lance. I have a fix out now (#198) that fixes that issue and another issue that would have been there due to the URL's in certification notes.

My question now is: If I use the checked-in SubmissionData.json file, how does Update-ApplicationSubmission know to use the new APPX? Does it automatically change the packages list when I use -ReplacePackages flag?

Don't check-in the submission data. Only check-in the config file. You should generate the "submission package" (which outputs a .json and .zip file) each time you want to do a submission. You reference the .json and .zip file in Update-ApplicationSubmission, and if you specify -ReplacePackages, it will replace the packages on PartnerCenter with the ones that are defined in the .json file. You may want to review USAGE.md again for a bit more clarity on this step.

PS - Despite these issues in GitHub Actions (PS core?) StoreBroker is an absolute pleasure in user-mode locally.

Thanks for the feedback! If you like StoreBroker, you may be interested in looking at PowerShellForGitHub as well. It's next major release (0.15.0) should be happening this weekend and has a ton of improvements (including full pipeline support).

from storebroker.

I was just looking closely at the PR with the fix, good stuff! Now I understand why there were so many comments in the config.json file :D

You should generate the "submission package" (which outputs a .json and .zip file) each time you want to do a submission. You reference the .json and .zip file in Update-ApplicationSubmission, and if you specify -ReplacePackages, it will replace the packages on PartnerCenter with the ones that are defined in the .json file.

This is one of the key places that I was misunderstanding, I have read USAGE.md several times, but it gets a little confusing about where the ZIP file comes from. The submission data path is clear because you set the name with -OutName, but I'm still not sure what the file name is for the ZIP?

To illustrate the question using the snippet below, is the generated ZIP file path MyFolder\package.zip?

New-SubmissionPackage -OutPath 'MyFolder\' -OutName 'submission.json'

In any case, this thread is not the right place to discuss this, I'll keep trying and open a new Issue if I need to. Here's what I settled on for upcoming builds:

# Temp folder for all StoreBroker artifacts
$sbTempFolderPath = New-Item -Type Directory -Force -Path (Join-Path -Path 'D:\a\MediaFileManager\MediaFileManager\' -ChildPath 'SBTemp')

# Use the committed config file to generate a new submission.json file
$configFilePath = 'D:\a\MediaFileManager\MediaFileManager\.scripts\storeBrokerConfig.json'
New-SubmissionPackage -ConfigPath $configFilePath -AppxPath $appxUploadFilePath -OutPath $sbTempFolderPath -OutName 'submission.json'

# Identify the ZIP and submission.json file paths
$submissionDataPath = Join-Path -Path $sbTempFolderPath -ChildPath 'submission.json'
$submissionPackagePath = Join-Path -Path $sbTempFolderPath -ChildPath 'package.zip'

Update-ApplicationSubmission -ReplacePackages -AppId $appStoreId -SubmissionDataPath $submissionDataPath -PackagePath $submissionPackagePath -AutoCommit -Force

Maybe it will be useful to add a new doc page for complete examples (instead of partial snippets) so the reader can see the use in entirety? This usually helps make everything click (after being taken through the individual steps).

I'd be happy to contribute to it with GitHub Actions example.

from storebroker.

Is this planned to merge soon? I noticed you have 2 PRs open.

I have an opportunity to try out the Broker again in a workflow because my last manual publish has passed certification (I would like to give the Action a shot for real).

from storebroker.