Creating an array from a txt file in bash

flukeyLinux · 03-20-2020, 10:22 AM

Hi all,
I'm looking for some help to build an array from a text file that looks like this:

Project1

ansible

Project2
Project3
Project4

repo1

Project5

repo1
repo2
repo3
repo4
repo5
repo6
repo7

Project6

repo1
repo2
repo3
repo4
repo5
repo6
repo7
repo8
repo9
repo10

Project7

repo1
repo2
repo3
repo4

Project8

ops

Project9

delete_this_repo

Project10

model_service
c2r_service
puppetcode

So it looks like this:

Project1=(ansible)
Project2=()
Project3=()
Project4=(repo1)
Project5=(repo1 repo2 repo3 repo4 repo5 repo6 repo7)
... and so on.

Anything that does not have a "repo" can just be deleted. That could be done before or after. I tried looking into "declare" but I am not sure how to use it. Maybe there is something with awk I could use?

This data was in jq but I removed all of the double quotes and commas to get the data I needed to look like it does above. But maybe there is an easier way to do this using jq?

Any suggestions welcome!

Thanks in advance,
Fluke

boughtonp · 03-20-2020, 10:55 AM

Quote:

Originally Posted by flukeyLinux

This data was in jq but I removed all of the double quotes and commas to get the data I needed to look like it does above. But maybe there is an easier way to do this using jq?

That's very likely - I've not yet used jq, but it looks like you could use recurse to filter the data, then map to reformat each item.
https://stedolan.github.io/jq/manual...),recurse_down
https://stedolan.github.io/jq/manual...,map_values(x)

Turbocapitalist · 03-20-2020, 11:02 AM

What does the above look like in the original JSON encoded format?

BW-userx · 03-20-2020, 12:57 PM

this is purely dynamically being done. so you're going to have to come up with a way to keep track of each array created dynamically and keep track of there names in another array perhaps.

just using that "file" you posted.

Code:

#!/bin/bash

arr=''

while read a ; 
do
#echo $a
if [[ "${a,,}" =~ "project" ]] ;
then
#keep which project it is
	arr=$a
	#to add in the print out of each
	#array created, but not kept.
	#because it is being over written by
	#the next projectX in line within 
	#the 'data' file.
	echo "$arr"
	
	#if projects hits then the next 
	#step is to get the repo's if any.
elif [[ "${a,,}" =~ "repo" ]] ;
then
	
	echo "$a"
	arr+=("$a")
	
	
fi

done < data

output

Code:

$ ./sortme
Project1
Project2
Project3
Project4
repo1
Project5
repo1
repo2
repo3
repo4
repo5
repo6
repo7
Project6
repo1
repo2
repo3
repo4
repo5
repo6
repo7
repo8
repo9
repo10
Project7
repo1
repo2
repo3
repo4
Project8
Project9
delete_this_repo
Project10
Project10

it is just an idea, work in progress. but I'd think it is still all being done dynamically regardless of what tools you use.

flukeyLinux · 03-23-2020, 07:18 AM

Quote:

Originally Posted by BW-userx

this is purely dynamically being done. so you're going to have to come up with a way to keep track of each array created dynamically and keep track of there names in another array perhaps.

just using that "file" you posted.

Code:

#!/bin/bash

arr=''

while read a ; 
do
#echo $a
if [[ "${a,,}" =~ "project" ]] ;
then
#keep which project it is
	arr=$a
	#to add in the print out of each
	#array created, but not kept.
	#because it is being over written by
	#the next projectX in line within 
	#the 'data' file.
	echo "$arr"
	
	#if projects hits then the next 
	#step is to get the repo's if any.
elif [[ "${a,,}" =~ "repo" ]] ;
then
	
	echo "$a"
	arr+=("$a")
	
	
fi

done < data

output

Code:

$ ./sortme
Project1
Project2
Project3
Project4
repo1
Project5
repo1
repo2
repo3
repo4
repo5
repo6
repo7
Project6
repo1
repo2
repo3
repo4
repo5
repo6
repo7
repo8
repo9
repo10
Project7
repo1
repo2
repo3
repo4
Project8
Project9
delete_this_repo
Project10
Project10

it is just an idea, work in progress. but I'd think it is still all being done dynamically regardless of what tools you use.

This would do actually but most repo's don't have the word repo in it. I am going to try going the JQ route but if I don't get what I need, I will be coming back to this.

thanks very much.

flukeyLinux · 03-23-2020, 07:27 AM

Quote:

Originally Posted by Turbocapitalist

What does the above look like in the original JSON encoded format?

The projects look like this:

"key": "Project1",
"key": "Project2",
"key": "Project3",
"key": "Project4",
"key": "Project5",
"key": "Project6",
"key": "Project7",
"key": "Project8",
"key": "Project9",
"key": "Project10",

I strip out the 1t column with awk and remove any white space with tr -d.

I then use a for loop to get the repo's under every project with an API call

# Get repos for every project
for i in `cat projectkeys.txt`;
do echo $i;
curl -s -u $LOGIN "https://$SERVER/rest/api/1.0/projects/$i/repos?limit=10" -k | jq . | grep "slug";
done > projectkeys2.txt

I couldn't do this with quotes and commas so that is why I removed them.

regards,
fluke

Turbocapitalist · 03-23-2020, 07:35 AM

Quote:

Originally Posted by flukeyLinux

The projects look like this:

That's not JSON though. Is that really how the source data looks? Please provide an exact sample of the format, sanitized if necessary but retaining the format nonetheless.

flukeyLinux · 03-23-2020, 07:43 AM

Quote:

Originally Posted by Turbocapitalist

That's not JSON though. Is that really how the source data looks? Please provide an exact sample of the format, sanitized if necessary but retaining the format nonetheless.

true, I use grep to get what I need. Here is the full output of the API call

Code:

{
  "size": 1,
  "limit": 1,
  "isLastPage": false,
  "values": [
    {
      "key": "Project4",
      "id": 15785,
      "name": "repo1",
      "public": false,
      "type": "NORMAL",
      "links": {
        "self": [
          {
            "href": "https://10.196.20.79:9443/projects/Project4"
          }
        ]
      }
    }
  ],
  "start": 0,
  "nextPageStart": 1
}